2024-03-29T11:22:27.695 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:22:27.695 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --verbose --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/orig.config.yaml 2024-03-29T11:22:27.729 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:22:27.827 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622711' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLfB6oBFzuCD9lJafl5tdfIzT0/P1jWSao7uthDO+9x9yuQW/zyI15ZyOrti/J982iJuh3X09GkUwuH85bqbgg= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUFjPwIOVgPGNqn7AKOiSDnNg4zyeWjB81qwLpb6ObHbse3OpFmOyuwE9sNtnpAqwNhxHi5RyHAklknpOIBC7s= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:22:27.827 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:22:27.828 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:22:27.829 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:22:27.831 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:22:27.867 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:22:27.867 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:22:27.867 INFO:teuthology.packaging:ref: None 2024-03-29T11:22:27.867 INFO:teuthology.packaging:tag: None 2024-03-29T11:22:27.867 INFO:teuthology.packaging:branch: main 2024-03-29T11:22:27.868 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:22:27.868 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:22:28.130 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:22:28.131 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:22:28.138 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:22:28.138 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:22:28.145 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:22:28.158 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:22:28.164 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:22:28.189 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.360207', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLfB6oBFzuCD9lJafl5tdfIzT0/P1jWSao7uthDO+9x9yuQW/zyI15ZyOrti/J982iJuh3X09GkUwuH85bqbgg='} 2024-03-29T11:22:28.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.359037', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUFjPwIOVgPGNqn7AKOiSDnNg4zyeWjB81qwLpb6ObHbse3OpFmOyuwE9sNtnpAqwNhxHi5RyHAklknpOIBC7s='} 2024-03-29T11:22:28.210 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:22:28.216 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:22:28.216 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:22:28.217 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:22:28.287 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f272d2260d0>, signals=[15]) 2024-03-29T11:22:28.288 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:22:28.294 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:22:28.294 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:22:28.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:28.402 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:22:28.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:28.489 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:22:28.496 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-03-29T11:22:28.513 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-03-29T11:22:28.514 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:28.571 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:28.572 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:28.572 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:28.572 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-03-29T11:22:28.601 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-29T11:22:28.619 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-29T11:22:28.619 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-29T11:22:28.676 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-29T11:22:28.676 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-29T11:22:28.676 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-29T11:22:28.676 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:28.677 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-29T11:22:28.677 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:28.677 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:28.677 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:28.677 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:28.677 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:28.678 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:28.678 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:28.678 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:28.678 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:28.678 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-29T11:22:28.700 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:22:28.709 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:22:28.735 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:22:28.735 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:28.737 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:28.753 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:22:28.761 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:22:28.761 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:28.794 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:28.825 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:22:28.887 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:22:28.909 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:22:28.910 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:22:28.910 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:28.911 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:28.928 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-29T11:22:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:28.930 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-29T11:22:28.984 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:28.985 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-03-29T11:22:28.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-03-29T11:22:28.989 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:28.989 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2024-03-29T11:22:28.989 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-03-29T11:22:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:22:31.257 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:22:32.791 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-29T11:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-29T11:22:33.637 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T11:22:35.223 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-29T11:22:35.757 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-29T11:22:41.600 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.1 MB 00:02 2024-03-29T11:22:42.964 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-29T11:22:44.641 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T11:22:44.818 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-03-29T11:22:49.057 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-29T11:22:49.658 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T11:22:50.817 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-03-29T11:22:51.956 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:22:51.957 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:22:52.045 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:22:52.056 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:22:52.056 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:22:52.056 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:22:52.056 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:22:52.056 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-03-29T11:22:52.057 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:22:52.058 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 76 M 2024-03-29T11:22:52.058 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-03-29T11:22:52.058 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:22:52.476 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 459 kB/s | 192 kB 00:00 2024-03-29T11:22:53.926 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 8.1 MB/s | 15 MB 00:01 2024-03-29T11:22:54.601 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-03-29T11:22:54.851 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 12 MB/s | 28 MB 00:02 2024-03-29T11:22:54.854 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:54.854 INFO:teuthology.orchestra.run.smithi153.stdout:Total 27 MB/s | 76 MB 00:02 2024-03-29T11:22:54.855 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:22:55.209 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:22:55.209 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:22:56.052 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:22:56.052 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:22:58.365 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:22:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:59.043 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:01.864 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:02.004 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:06.736 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:06.777 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:11.523 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:31.281 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 527 kB/s | 21 MB 00:40 2024-03-29T11:23:36.366 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-29T11:23:38.721 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:23:38.721 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:23:38.811 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:23:38.816 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2024-03-29T11:23:38.817 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:23:38.818 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 76 M 2024-03-29T11:23:38.818 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 123 M 2024-03-29T11:23:38.818 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:23:38.974 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:23:39.374 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 27 MB/s | 15 MB 00:00 2024-03-29T11:23:40.024 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T11:23:40.358 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T11:23:40.361 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:40.361 INFO:teuthology.orchestra.run.smithi033.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-29T11:23:40.361 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:23:40.778 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:23:40.778 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:23:40.796 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:23:40.796 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:23:40.821 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:41.211 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:41.212 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:41.212 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:41.212 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:41.596 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:41.596 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:23:41.597 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:23:41.597 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:41.597 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:41.597 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:23:41.597 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:41.597 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:23:41.597 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:23:41.688 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:23:41.688 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:23:41.931 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:42.027 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:42.027 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:42.028 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:23:42.028 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:23:42.028 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:23:42.028 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:23:42.028 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:23:42.028 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:42.107 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:42.108 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:42.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-03-29T11:23:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:23:42.147 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-03-29T11:23:42.148 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:23:42.148 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:23:42.148 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:23:42.148 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:23:42.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:23:42.743 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-29T11:23:44.109 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:23:44.202 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:23:44.223 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-29T11:23:44.225 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-29T11:23:44.240 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.cZiiTqRRj3 2024-03-29T11:23:44.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cZiiTqRRj3 2024-03-29T11:23:44.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.cZiiTqRRj3 2024-03-29T11:23:44.452 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.cZiiTqRRj3 is 6KB 2024-03-29T11:23:44.470 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.cZiiTqRRj3 2024-03-29T11:23:44.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:23:44.549 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:23:44.549 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:23:44.549 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:23:44.550 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:23:44.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:23:44.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-29T11:23:44.729 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:44.816 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:47.729 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:47.869 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:52.981 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:53.018 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:58.149 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:14.713 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:24:14.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-29T11:24:14.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:28.919 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:28.920 INFO:teuthology.orchestra.run.smithi033.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:24:28.920 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:24:28.963 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:29.349 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:29.349 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:29.349 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:29.349 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:29.756 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:24:29.757 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:24:30.088 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:30.186 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:30.187 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:30.187 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:24:30.187 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:24:30.187 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:24:30.187 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:24:30.187 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:24:30.187 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:30.271 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:30.271 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:30.271 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2024-03-29T11:24:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-03-29T11:24:30.313 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:24:30.314 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:24:30.314 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:24:30.314 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:24:30.315 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:24:30.841 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-03-29T11:24:32.239 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:24:32.296 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-03-29T11:24:32.298 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-29T11:24:32.314 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.tzz3Gh5IUJ 2024-03-29T11:24:32.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tzz3Gh5IUJ 2024-03-29T11:24:32.400 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.tzz3Gh5IUJ 2024-03-29T11:24:32.502 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.tzz3Gh5IUJ is 6KB 2024-03-29T11:24:32.551 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.tzz3Gh5IUJ 2024-03-29T11:24:32.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:24:32.635 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:24:32.635 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:24:32.635 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:24:32.635 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:24:32.637 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:24:32.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-03-29T11:24:49.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-29T11:24:58.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-29T11:24:58.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:02.782 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:25:02.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-29T11:25:02.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:21.311 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-29T11:25:30.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-29T11:25:30.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:48.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-29T11:25:58.328 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:26:00.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-29T11:26:00.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:02.820 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-29T11:26:03.271 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-29T11:26:03.271 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:03.271 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-29T11:26:03.287 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:03.288 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:03.288 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:03.288 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:04.288 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:26:04.289 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:04.410 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-03-29T11:26:04.454 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:10.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-29T11:26:10.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:10.717 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-29T11:26:11.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-29T11:26:11.117 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:11.117 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-29T11:26:11.132 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:11.132 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:11.132 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:11.132 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:12.133 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:26:12.134 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:12.217 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-29T11:26:12.250 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:12.250 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:26:12.258 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:26:12.258 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:12.261 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:12.279 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:26:12.285 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:26:12.292 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:26:12.292 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:12.321 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:12.347 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:26:12.354 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:26:12.354 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:12.386 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:12.412 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:12.425 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:12.428 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:12.442 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:12.443 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:26:12.450 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:26:12.450 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:12.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:12.514 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:26:12.522 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:26:12.523 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:12.544 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:12.570 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:12.641 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:12.704 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:12.789 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:12.835 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:26:12.835 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:12.905 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:12.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:13.007 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:13.092 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:13.138 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:26:13.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:13.207 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-03-29T11:26:13.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-29T11:26:13.253 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:13.812 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:26:13.820 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:26:13.821 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:26:13.831 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:26:13.841 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-03-29T11:26:13.921 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-03-29T11:26:13.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-29T11:26:13.979 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-29T11:26:13.981 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:26:13.982 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-03-29T11:26:14.006 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-03-29T11:26:14.006 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-29T11:26:14.028 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-29T11:26:14.028 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:26:14.028 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:14.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:14.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:14.108 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:26:14.108 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:14.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:14.188 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:26:14.198 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:26:14.215 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:26:14.217 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:30:36.558 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-03-29T11:30:36.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-29T11:30:36.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:36.662 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-29T11:30:36.745 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-29T11:30:36.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-29T11:30:36.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:36.834 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-29T11:30:36.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-29T11:30:36.910 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:30:36.922 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:30:36.922 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:36.923 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:36.925 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:36.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:36.954 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:36.972 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:36.999 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:37.001 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-03-29T11:30:37.015 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-03-29T11:30:37.029 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-29T11:30:37.032 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:37.043 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-29T11:30:37.048 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:37.060 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:37.076 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:37.106 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:37.106 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:37.107 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-29T11:30:37.107 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.107 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.107 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.107 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.128 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:37.131 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:37.131 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-29T11:30:37.131 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.131 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.131 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.131 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:37.132 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:30:37.140 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:30:37.141 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:30:37.141 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:30:37.141 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:30:37.146 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:30:37.147 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:30:37.147 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:37.147 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:37.149 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:37.150 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:37.291 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:37.292 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:37.371 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:37.371 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:37.463 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:37.464 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:30:37.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:37.484 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:37.484 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:30:37.485 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:37.492 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:37.493 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:37.512 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:37.513 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:37.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:37.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:37.640 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:37.660 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:37.706 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-03-29T11:30:37.708 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-03-29T11:30:37.727 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-29T11:30:37.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-29T11:30:37.964 INFO:teuthology.orchestra.run.smithi033.stdout:74 files removed 2024-03-29T11:30:37.976 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-29T11:30:37.992 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:38.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:38.787 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 167 kB/s | 82 kB 00:00 2024-03-29T11:30:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 166 kB/s | 82 kB 00:00 2024-03-29T11:30:39.297 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-29T11:30:39.379 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-29T11:30:39.856 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-29T11:30:40.147 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-29T11:30:40.508 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T11:30:41.717 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 4.4 MB/s | 6.7 MB 00:01 2024-03-29T11:30:42.575 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 2.1 MB/s | 2.0 MB 00:00 2024-03-29T11:30:43.524 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-29T11:30:44.982 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-29T11:30:48.102 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-03-29T11:30:52.410 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-29T11:30:52.939 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 908 kB/s | 8.0 MB 00:09 2024-03-29T11:30:54.019 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-29T11:30:54.874 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-29T11:30:55.914 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T11:30:59.835 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.1 MB 00:01 2024-03-29T11:31:00.878 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-29T11:31:01.317 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-03-29T11:31:03.385 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:31:03.499 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:31:03.508 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:31:03.508 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:31:03.508 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:31:03.508 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:31:03.508 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:03.509 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:03.510 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:31:03.511 INFO:teuthology.orchestra.run.smithi153.stdout:Install 30 Packages 2024-03-29T11:31:03.512 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-03-29T11:31:03.512 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:03.512 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-03-29T11:31:03.512 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:31:05.679 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.5 MB/s | 5.3 MB 00:02 2024-03-29T11:31:05.738 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 25 kB 00:00 2024-03-29T11:31:06.055 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 8.0 MB/s | 20 MB 00:02 2024-03-29T11:31:06.097 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.0 MB/s | 742 kB 00:00 2024-03-29T11:31:06.122 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:31:06.239 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 3.6 MB/s | 9.7 MB 00:02 2024-03-29T11:31:06.264 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 317 kB/s | 45 kB 00:00 2024-03-29T11:31:06.306 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T11:31:06.323 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T11:31:06.373 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.7 MB/s | 320 kB 00:00 2024-03-29T11:31:06.398 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T11:31:06.457 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T11:31:06.516 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 905 kB/s | 106 kB 00:00 2024-03-29T11:31:06.541 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T11:31:06.566 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-29T11:31:06.609 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:31:06.633 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-29T11:31:06.659 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-03-29T11:31:06.684 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:31:06.709 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 700 kB/s | 35 kB 00:00 2024-03-29T11:31:06.735 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T11:31:06.869 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 6.6 MB/s | 5.1 MB 00:00 2024-03-29T11:31:07.294 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 69 kB/s | 26 kB 00:00 2024-03-29T11:31:07.419 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 435 kB/s | 309 kB 00:00 2024-03-29T11:31:07.469 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2024-03-29T11:31:07.528 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-29T11:31:07.678 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:31:07.814 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2024-03-29T11:31:07.870 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-29T11:31:08.046 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.3 MB/s | 1.6 MB 00:00 2024-03-29T11:31:08.436 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-29T11:31:08.937 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.0 MB/s | 3.4 MB 00:01 2024-03-29T11:31:09.213 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 3.1 MB 00:01 2024-03-29T11:31:09.219 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:09.219 INFO:teuthology.orchestra.run.smithi153.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-29T11:31:09.219 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:31:09.257 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:31:09.257 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:31:09.737 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:31:09.737 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:31:11.178 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:31:11.193 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:11.198 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:11.199 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:11.200 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout:Install 30 Packages 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 2 Packages 2024-03-29T11:31:11.201 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:11.202 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 58 M 2024-03-29T11:31:11.202 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:31:11.273 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:11.301 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:11.344 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:11.366 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:11.383 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:11.575 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:11.629 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:11.948 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:12.206 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:12.246 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:12.317 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:12.346 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:12.389 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:12.457 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:12.482 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:12.545 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:12.581 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:12.732 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:12.792 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:12.821 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:12.851 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:12.883 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:13.209 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:13.246 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:13.328 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:13.360 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:13.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:13.877 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:13.907 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:13.962 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:14.030 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:14.059 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:16.248 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:16.298 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:16.661 INFO:teuthology.orchestra.run.smithi033.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 5.3 MB 00:05 2024-03-29T11:31:16.820 INFO:teuthology.orchestra.run.smithi033.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 159 kB/s | 25 kB 00:00 2024-03-29T11:31:16.842 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:16.876 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:16.950 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:16.950 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:16.986 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:18.054 INFO:teuthology.orchestra.run.smithi033.stdout:(3/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 9.7 MB 00:06 2024-03-29T11:31:18.078 INFO:teuthology.orchestra.run.smithi033.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 589 kB/s | 742 kB 00:01 2024-03-29T11:31:18.554 INFO:teuthology.orchestra.run.smithi033.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 996 kB/s | 498 kB 00:00 2024-03-29T11:31:18.688 INFO:teuthology.orchestra.run.smithi033.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 338 kB/s | 45 kB 00:00 2024-03-29T11:31:18.888 INFO:teuthology.orchestra.run.smithi033.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 657 kB/s | 131 kB 00:00 2024-03-29T11:31:19.046 INFO:teuthology.orchestra.run.smithi033.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.0 MB/s | 162 kB 00:00 2024-03-29T11:31:19.313 INFO:teuthology.orchestra.run.smithi033.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.2 MB/s | 320 kB 00:00 2024-03-29T11:31:19.589 INFO:teuthology.orchestra.run.smithi033.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 298 kB 00:00 2024-03-29T11:31:19.706 INFO:teuthology.orchestra.run.smithi033.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 851 kB/s | 99 kB 00:00 2024-03-29T11:31:19.814 INFO:teuthology.orchestra.run.smithi033.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-29T11:31:19.881 INFO:teuthology.orchestra.run.smithi033.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-29T11:31:19.915 INFO:teuthology.orchestra.run.smithi033.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:31:19.965 INFO:teuthology.orchestra.run.smithi033.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T11:31:20.007 INFO:teuthology.orchestra.run.smithi033.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:31:20.041 INFO:teuthology.orchestra.run.smithi033.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:31:20.074 INFO:teuthology.orchestra.run.smithi033.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.4 MB/s | 220 kB 00:00 2024-03-29T11:31:20.110 INFO:teuthology.orchestra.run.smithi033.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:31:20.142 INFO:teuthology.orchestra.run.smithi033.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:31:20.567 INFO:teuthology.orchestra.run.smithi033.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 727 kB/s | 309 kB 00:00 2024-03-29T11:31:20.884 INFO:teuthology.orchestra.run.smithi033.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-29T11:31:20.959 INFO:teuthology.orchestra.run.smithi033.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-29T11:31:21.035 INFO:teuthology.orchestra.run.smithi033.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 649 kB/s | 49 kB 00:00 2024-03-29T11:31:21.101 INFO:teuthology.orchestra.run.smithi033.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:31:21.193 INFO:teuthology.orchestra.run.smithi033.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-29T11:31:21.269 INFO:teuthology.orchestra.run.smithi033.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-29T11:31:21.369 INFO:teuthology.orchestra.run.smithi033.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-29T11:31:23.586 INFO:teuthology.orchestra.run.smithi033.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.6 MB/s | 20 MB 00:12 2024-03-29T11:31:24.002 INFO:teuthology.orchestra.run.smithi033.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 3.4 MB 00:02 2024-03-29T11:31:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:(31/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 812 kB/s | 5.1 MB 00:06 2024-03-29T11:31:26.655 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:26.655 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-29T11:31:26.655 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-29T11:31:26.656 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-29T11:31:26.656 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-29T11:31:26.656 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-29T11:31:26.656 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-29T11:31:26.656 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-29T11:31:26.656 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:27.178 INFO:teuthology.orchestra.run.smithi033.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 892 kB/s | 3.1 MB 00:03 2024-03-29T11:31:27.182 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:27.182 INFO:teuthology.orchestra.run.smithi033.stdout:Total 3.6 MB/s | 58 MB 00:15 2024-03-29T11:31:27.183 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:31:27.225 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:31:27.225 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:31:27.533 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:27.555 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:27.555 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:27.555 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:31:27.555 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:27.556 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:27.556 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:27.557 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:27.636 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:27.704 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:31:27.708 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:31:28.837 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:28.837 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:31:28.837 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:31:28.837 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:31:28.837 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:31:28.837 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:31:28.841 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:31:28.842 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:31:28.843 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:28.843 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:31:28.843 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:31:29.355 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:31:29.432 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:29.466 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:29.495 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:29.523 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:29.538 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:29.734 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:29.787 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:29.918 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:31:30.131 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:30.193 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:30.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-03-29T11:31:30.285 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:30.476 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:30.512 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:30.574 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:30.604 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:30.657 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:30.742 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:30.772 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:30.828 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:30.838 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:31:30.862 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:30.974 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:30.999 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:31:31.000 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:31:31.000 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:31:31.000 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:31:31.000 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-29T11:31:31.001 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:31:31.002 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 48 M 2024-03-29T11:31:31.002 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 204 M 2024-03-29T11:31:31.002 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:31:31.005 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:31.046 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:31.079 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:31.107 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:31.140 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:31.446 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 816 kB/s | 217 kB 00:00 2024-03-29T11:31:31.477 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:31.512 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:31.588 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-03-29T11:31:31.602 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:31.638 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:31:31.654 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:31.705 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:31.880 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-03-29T11:31:31.989 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 163 kB 00:00 2024-03-29T11:31:32.184 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:32.215 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:32.269 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:32.340 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:32.367 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:34.654 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:34.697 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:35.275 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:35.309 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:35.405 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:35.405 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-03-29T11:31:45.349 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-03-29T11:31:45.350 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:46.282 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:46.305 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:46.305 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:46.305 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:31:46.305 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:46.306 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:46.306 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:46.307 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:46.401 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:46.401 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:31:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:31:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:31:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:31:48.478 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:48.478 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:48.478 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-03-29T11:31:48.478 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.478 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.478 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:31:48.479 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:31:48.480 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:31:48.481 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:31:48.481 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:48.481 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:31:48.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-test 2024-03-29T11:31:49.351 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:31:49.507 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:31:49.507 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-03-29T11:31:49.509 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:31:49.509 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 48 M 2024-03-29T11:31:49.509 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 204 M 2024-03-29T11:31:49.509 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:31:49.912 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-29T11:31:49.979 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.9 MB/s | 300 kB 00:00 2024-03-29T11:31:50.012 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:31:50.146 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:31:50.413 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 258 kB/s | 163 kB 00:00 2024-03-29T11:32:08.102 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 47 MB 00:36 2024-03-29T11:32:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.3 MB/s | 48 MB 00:37 2024-03-29T11:32:08.117 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:32:08.137 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:32:08.137 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:32:08.527 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:32:08.528 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:32:09.103 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:32:09.169 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:09.217 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:09.256 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:09.294 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:09.322 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:13.801 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:13.888 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:16.284 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:16.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:16.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:16.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:16.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:16.285 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:16.418 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 47 MB 00:26 2024-03-29T11:32:16.421 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:16.422 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.8 MB/s | 48 MB 00:26 2024-03-29T11:32:16.436 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:32:16.460 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:32:16.460 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:32:16.652 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:16.652 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:16.652 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:32:16.652 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:16.652 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:16.652 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:16.653 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:16.653 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:16.653 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:16.653 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:16.653 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:32:16.853 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:32:16.853 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:32:16.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-03-29T11:32:17.441 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:32:17.512 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:17.526 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:32:17.552 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:17.593 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:17.637 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:17.663 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:17.683 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:17.686 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout:Install 36 Packages 2024-03-29T11:32:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:17.690 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 30 M 2024-03-29T11:32:17.690 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 107 M 2024-03-29T11:32:17.690 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:32:18.822 INFO:teuthology.orchestra.run.smithi153.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-29T11:32:20.198 INFO:teuthology.orchestra.run.smithi153.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 852 kB/s | 1.5 MB 00:01 2024-03-29T11:32:20.881 INFO:teuthology.orchestra.run.smithi153.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 879 kB/s | 2.1 MB 00:02 2024-03-29T11:32:21.181 INFO:teuthology.orchestra.run.smithi153.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 825 kB/s | 247 kB 00:00 2024-03-29T11:32:21.298 INFO:teuthology.orchestra.run.smithi153.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:32:21.345 INFO:teuthology.orchestra.run.smithi153.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.0 MB/s | 144 kB 00:00 2024-03-29T11:32:21.382 INFO:teuthology.orchestra.run.smithi153.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 867 kB/s | 32 kB 00:00 2024-03-29T11:32:21.416 INFO:teuthology.orchestra.run.smithi153.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:32:21.449 INFO:teuthology.orchestra.run.smithi153.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:32:21.491 INFO:teuthology.orchestra.run.smithi153.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:32:21.566 INFO:teuthology.orchestra.run.smithi153.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:32:21.600 INFO:teuthology.orchestra.run.smithi153.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:32:21.634 INFO:teuthology.orchestra.run.smithi153.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-29T11:32:21.668 INFO:teuthology.orchestra.run.smithi153.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:32:21.734 INFO:teuthology.orchestra.run.smithi153.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-03-29T11:32:21.867 INFO:teuthology.orchestra.run.smithi153.stdout:(16/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 4.7 MB 00:03 2024-03-29T11:32:21.893 INFO:teuthology.orchestra.run.smithi153.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-03-29T11:32:22.068 INFO:teuthology.orchestra.run.smithi153.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 752 kB/s | 151 kB 00:00 2024-03-29T11:32:22.096 INFO:teuthology.orchestra.run.smithi153.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 217 kB/s | 43 kB 00:00 2024-03-29T11:32:22.160 INFO:teuthology.orchestra.run.smithi153.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-29T11:32:22.186 INFO:teuthology.orchestra.run.smithi153.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:32:22.203 INFO:teuthology.orchestra.run.smithi153.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-29T11:32:22.220 INFO:teuthology.orchestra.run.smithi153.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 530 kB/s | 18 kB 00:00 2024-03-29T11:32:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 689 kB/s | 23 kB 00:00 2024-03-29T11:32:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 462 kB/s | 19 kB 00:00 2024-03-29T11:32:22.278 INFO:teuthology.orchestra.run.smithi153.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 471 kB/s | 20 kB 00:00 2024-03-29T11:32:22.305 INFO:teuthology.orchestra.run.smithi153.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-29T11:32:22.321 INFO:teuthology.orchestra.run.smithi153.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-29T11:32:22.355 INFO:teuthology.orchestra.run.smithi153.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T11:32:22.388 INFO:teuthology.orchestra.run.smithi153.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-29T11:32:22.389 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:22.414 INFO:teuthology.orchestra.run.smithi153.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T11:32:22.439 INFO:teuthology.orchestra.run.smithi153.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-03-29T11:32:22.465 INFO:teuthology.orchestra.run.smithi153.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T11:32:22.467 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:22.491 INFO:teuthology.orchestra.run.smithi153.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.3 MB/s | 427 kB 00:00 2024-03-29T11:32:22.515 INFO:teuthology.orchestra.run.smithi153.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 406 kB/s | 20 kB 00:00 2024-03-29T11:32:24.820 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:24.820 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:24.821 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:24.821 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:24.821 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:24.821 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:25.168 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:25.168 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:25.169 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:32:25.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph 2024-03-29T11:32:26.008 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:32:26.168 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:26.171 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:26.172 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:26.173 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:26.174 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:26.174 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:32:26.174 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-29T11:32:26.174 INFO:teuthology.orchestra.run.smithi033.stdout:Install 36 Packages 2024-03-29T11:32:26.174 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:26.174 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 30 M 2024-03-29T11:32:26.174 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 107 M 2024-03-29T11:32:26.175 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:32:27.371 INFO:teuthology.orchestra.run.smithi033.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-29T11:32:30.112 INFO:teuthology.orchestra.run.smithi033.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 477 kB/s | 1.5 MB 00:03 2024-03-29T11:32:31.754 INFO:teuthology.orchestra.run.smithi033.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 449 kB/s | 2.1 MB 00:04 2024-03-29T11:32:32.597 INFO:teuthology.orchestra.run.smithi033.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 294 kB/s | 247 kB 00:00 2024-03-29T11:32:32.821 INFO:teuthology.orchestra.run.smithi033.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-03-29T11:32:32.866 INFO:teuthology.orchestra.run.smithi033.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:32:32.899 INFO:teuthology.orchestra.run.smithi033.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 944 kB/s | 32 kB 00:00 2024-03-29T11:32:32.931 INFO:teuthology.orchestra.run.smithi033.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:32:32.964 INFO:teuthology.orchestra.run.smithi033.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:32:33.014 INFO:teuthology.orchestra.run.smithi033.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:32:33.082 INFO:teuthology.orchestra.run.smithi033.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:32:33.115 INFO:teuthology.orchestra.run.smithi033.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:32:33.157 INFO:teuthology.orchestra.run.smithi033.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T11:32:33.191 INFO:teuthology.orchestra.run.smithi033.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:32:33.416 INFO:teuthology.orchestra.run.smithi033.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 555 kB/s | 125 kB 00:00 2024-03-29T11:32:33.499 INFO:teuthology.orchestra.run.smithi033.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-29T11:32:33.810 INFO:teuthology.orchestra.run.smithi033.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 745 kB/s | 4.7 MB 00:06 2024-03-29T11:32:34.008 INFO:teuthology.orchestra.run.smithi033.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 297 kB/s | 151 kB 00:00 2024-03-29T11:32:34.159 INFO:teuthology.orchestra.run.smithi033.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-29T11:32:34.451 INFO:teuthology.orchestra.run.smithi033.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 389 kB/s | 172 kB 00:00 2024-03-29T11:32:34.634 INFO:teuthology.orchestra.run.smithi033.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-03-29T11:32:34.659 INFO:teuthology.orchestra.run.smithi033.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 716 kB/s | 358 kB 00:00 2024-03-29T11:32:34.901 INFO:teuthology.orchestra.run.smithi033.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-29T11:32:34.926 INFO:teuthology.orchestra.run.smithi033.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-03-29T11:32:35.168 INFO:teuthology.orchestra.run.smithi033.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 73 kB/s | 19 kB 00:00 2024-03-29T11:32:35.194 INFO:teuthology.orchestra.run.smithi033.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 74 kB/s | 20 kB 00:00 2024-03-29T11:32:35.519 INFO:teuthology.orchestra.run.smithi033.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 132 kB/s | 46 kB 00:00 2024-03-29T11:32:35.536 INFO:teuthology.orchestra.run.smithi033.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 230 kB/s | 79 kB 00:00 2024-03-29T11:32:35.803 INFO:teuthology.orchestra.run.smithi033.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-03-29T11:32:36.028 INFO:teuthology.orchestra.run.smithi033.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 535 kB/s | 272 kB 00:00 2024-03-29T11:32:36.229 INFO:teuthology.orchestra.run.smithi033.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 211 kB/s | 90 kB 00:00 2024-03-29T11:32:36.296 INFO:teuthology.orchestra.run.smithi033.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-29T11:32:36.737 INFO:teuthology.orchestra.run.smithi033.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-03-29T11:32:36.813 INFO:teuthology.orchestra.run.smithi153.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 17 MB 00:16 2024-03-29T11:32:36.819 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:36.819 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.6 MB/s | 30 MB 00:19 2024-03-29T11:32:36.843 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:32:36.886 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:32:36.886 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:32:36.888 INFO:teuthology.orchestra.run.smithi033.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 722 kB/s | 427 kB 00:00 2024-03-29T11:32:37.005 INFO:teuthology.orchestra.run.smithi033.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-03-29T11:32:37.307 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:32:37.307 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:32:38.624 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:32:38.682 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:32:38.726 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:32:38.760 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:32:38.836 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:32:38.866 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:32:38.899 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:32:38.992 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:32:39.077 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:32:39.118 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:32:39.191 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:32:39.245 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:32:39.283 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:32:39.331 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:32:39.387 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:32:39.572 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:32:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:32:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:32:39.693 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:32:39.723 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:32:39.799 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:32:39.834 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:39.867 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:32:39.907 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:32:39.942 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:32:39.985 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:32:40.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:32:40.115 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:32:40.189 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:32:40.326 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:32:40.450 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:40.607 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:32:40.637 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:40.663 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:40.663 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:40.663 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:32:40.663 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:40.663 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:40.663 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:42.104 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:42.131 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:42.131 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:42.131 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:32:42.131 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:42.131 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:42.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:42.571 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:42.595 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:42.595 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:42.595 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:32:42.595 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:42.595 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:42.595 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:42.806 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:42.829 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:42.829 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:42.829 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:32:42.829 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:42.829 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:42.829 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:42.895 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:44.638 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:32:44.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:32:44.640 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:32:44.641 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:44.641 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:32:44.641 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:32:44.641 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:32:44.641 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:32:44.641 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:32:45.637 INFO:teuthology.orchestra.run.smithi033.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 17 MB 00:15 2024-03-29T11:32:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:45.641 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.5 MB/s | 30 MB 00:19 2024-03-29T11:32:45.665 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:32:45.711 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:32:45.711 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.723 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:32:45.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:32:45.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:32:45.726 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:45.726 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:32:46.128 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:32:46.128 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:32:46.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-03-29T11:32:46.736 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:32:46.841 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:46.898 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:32:46.899 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:32:46.899 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:32:46.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-03-29T11:32:47.476 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:32:47.548 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:32:47.578 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:32:47.581 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:32:47.608 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:32:47.683 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:32:47.725 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:32:47.736 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:32:47.737 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:32:47.737 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:32:47.737 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 747 k 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 754 k 2024-03-29T11:32:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:32:47.757 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:32:47.852 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:32:47.930 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:32:47.979 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:32:48.288 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:32:48.288 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:32:48.288 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:32:48.288 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:32:48.289 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:32:48.291 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.4 MB/s | 747 kB 00:00 2024-03-29T11:32:48.291 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:48.291 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.4 MB/s | 747 kB 00:00 2024-03-29T11:32:48.291 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:32:48.291 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:32:48.291 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:32:48.291 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:32:48.292 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:32:48.292 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:32:48.458 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:32:48.478 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:48.503 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:32:48.542 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:32:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:32:48.606 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:32:48.685 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:32:48.725 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:48.758 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:32:48.799 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:32:48.843 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:32:48.877 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:32:48.970 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:32:48.991 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:49.016 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:32:49.091 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:32:49.239 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:32:49.276 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:49.276 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:49.276 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:32:49.276 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:49.277 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:49.277 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:32:49.374 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:49.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:32:49.540 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:32:49.570 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:49.595 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:32:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:49.595 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:49.595 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:50.045 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:32:50.200 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:32:50.200 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:32:50.200 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-29T11:32:50.201 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:50.202 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 145 k 2024-03-29T11:32:50.202 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 436 k 2024-03-29T11:32:50.202 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:32:50.560 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 406 kB/s | 145 kB 00:00 2024-03-29T11:32:50.560 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:50.561 INFO:teuthology.orchestra.run.smithi153.stdout:Total 404 kB/s | 145 kB 00:00 2024-03-29T11:32:50.561 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:32:50.569 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:32:50.622 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:32:50.622 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:32:50.759 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:32:50.819 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:50.844 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:50.844 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:32:50.844 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:51.113 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:51.139 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:51.139 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:51.139 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:32:51.139 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:51.139 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:51.139 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:51.597 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:51.621 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:51.838 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:51.838 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:51.838 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:32:51.839 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:51.839 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:51.839 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:32:51.839 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:51.862 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:51.862 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:51.862 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:32:51.863 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:51.863 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:51.863 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:51.931 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:52.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-03-29T11:32:52.680 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:32:52.780 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:52.832 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:32:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:32:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:32:52.905 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:32:53.472 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:32:53.623 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:32:53.625 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-29T11:32:53.626 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:53.627 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.9 M 2024-03-29T11:32:53.627 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-29T11:32:53.627 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:32:53.659 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:53.659 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:32:53.659 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:32:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:32:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:32:53.662 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:53.662 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:32:53.662 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:32:53.662 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:32:53.662 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:32:53.662 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:32:53.997 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 75 kB/s | 15 kB 00:00 2024-03-29T11:32:54.014 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 142 kB/s | 31 kB 00:00 2024-03-29T11:32:54.173 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-29T11:32:54.231 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-29T11:32:54.365 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-29T11:32:54.490 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-29T11:32:54.524 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:54.544 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:32:54.545 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:32:54.546 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:54.547 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:32:54.582 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-29T11:32:54.641 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 526 kB/s | 31 kB 00:00 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T11:32:54.818 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.3 MB/s | 2.0 MB 00:00 2024-03-29T11:32:54.858 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.7 MB/s | 1.8 MB 00:01 2024-03-29T11:32:54.862 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:54.862 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.5 MB/s | 6.9 MB 00:01 2024-03-29T11:32:54.880 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:32:54.897 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:32:54.897 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:32:54.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-base 2024-03-29T11:32:55.068 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:32:55.068 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:32:55.569 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:32:55.574 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:32:55.651 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:32:55.671 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:55.685 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:32:55.726 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:32:55.727 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:32:55.728 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:32:55.732 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:32:55.762 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:32:55.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install cephadm 2024-03-29T11:32:55.867 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:32:55.989 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:32:56.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:32:56.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:32:56.170 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:56.202 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:32:56.360 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:32:56.445 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:32:56.477 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:32:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 747 k 2024-03-29T11:32:56.520 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 754 k 2024-03-29T11:32:56.520 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:32:57.276 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:57.276 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:32:57.277 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:32:57.623 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:32:57.624 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:57.625 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:32:57.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:32:58.475 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:32:58.631 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:32:58.632 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-29T11:32:58.632 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:32:58.632 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-29T11:32:58.632 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:32:58.632 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:32:58.633 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-29T11:32:58.634 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-29T11:32:58.634 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:32:58.634 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-03-29T11:32:58.634 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-03-29T11:32:58.634 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:32:58.746 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 336 kB/s | 747 kB 00:02 2024-03-29T11:32:58.746 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:58.747 INFO:teuthology.orchestra.run.smithi033.stdout:Total 336 kB/s | 747 kB 00:02 2024-03-29T11:32:58.747 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:32:58.753 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:32:58.754 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:32:58.760 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:32:58.760 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:32:58.836 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:32:59.065 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 182 kB/s | 32 kB 00:00 2024-03-29T11:32:59.098 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2024-03-29T11:32:59.132 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:32:59.232 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.8 MB/s | 3.0 MB 00:00 2024-03-29T11:32:59.308 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:59.403 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:59.466 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-29T11:32:59.533 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-29T11:32:59.558 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:32:59.776 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:59.908 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-29T11:33:00.008 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-29T11:33:00.056 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:00.056 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:00.056 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:00.056 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:00.056 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:00.056 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:00.100 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-29T11:33:00.167 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-29T11:33:00.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:33:00.826 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:00.986 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 145 k 2024-03-29T11:33:00.987 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 436 k 2024-03-29T11:33:00.988 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:01.542 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 262 kB/s | 145 kB 00:00 2024-03-29T11:33:01.543 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:01.543 INFO:teuthology.orchestra.run.smithi033.stdout:Total 261 kB/s | 145 kB 00:00 2024-03-29T11:33:01.543 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:01.552 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:01.552 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:01.608 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:01.608 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:01.752 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:01.810 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:01.834 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:01.835 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:01.835 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:33:01.835 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:02.740 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:02.740 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:02.740 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:02.741 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:02.741 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:02.741 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:02.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr 2024-03-29T11:33:03.562 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:03.664 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:03.719 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:03.721 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:03.721 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:03.785 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:33:04.357 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:04.518 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:33:04.519 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:33:04.520 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:33:04.520 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:33:04.520 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:33:04.520 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:33:04.520 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:33:04.520 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:33:04.520 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:33:04.521 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:33:04.521 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:04.521 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:04.521 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-29T11:33:04.521 INFO:teuthology.orchestra.run.smithi033.stdout:Install 12 Packages 2024-03-29T11:33:04.521 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:04.522 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 6.9 M 2024-03-29T11:33:04.522 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 26 M 2024-03-29T11:33:04.522 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:05.196 INFO:teuthology.orchestra.run.smithi033.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 89 kB/s | 31 kB 00:00 2024-03-29T11:33:05.255 INFO:teuthology.orchestra.run.smithi033.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 37 kB/s | 15 kB 00:00 2024-03-29T11:33:05.355 INFO:teuthology.orchestra.run.smithi033.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:33:05.380 INFO:teuthology.orchestra.run.smithi033.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-29T11:33:05.780 INFO:teuthology.orchestra.run.smithi033.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-03-29T11:33:05.805 INFO:teuthology.orchestra.run.smithi033.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-29T11:33:06.006 INFO:teuthology.orchestra.run.smithi033.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 10 MB/s | 2.0 MB 00:00 2024-03-29T11:33:06.031 INFO:teuthology.orchestra.run.smithi033.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 77 kB/s | 19 kB 00:00 2024-03-29T11:33:06.074 INFO:teuthology.orchestra.run.smithi033.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-29T11:33:06.132 INFO:teuthology.orchestra.run.smithi033.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 307 kB/s | 31 kB 00:00 2024-03-29T11:33:06.157 INFO:teuthology.orchestra.run.smithi033.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:33:06.965 INFO:teuthology.orchestra.run.smithi033.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 860 kB/s | 1.8 MB 00:02 2024-03-29T11:33:06.971 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:06.971 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-29T11:33:06.992 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:07.012 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:07.012 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:07.205 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:07.205 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:07.720 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:07.758 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 853 kB/s | 7.4 MB 00:08 2024-03-29T11:33:07.760 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:07.761 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.1 MB/s | 37 MB 00:09 2024-03-29T11:33:07.800 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:07.804 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:07.830 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:07.831 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:07.835 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:07.883 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:07.920 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:08.021 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:08.151 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:08.193 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:08.194 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:08.246 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:08.292 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:08.320 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:08.351 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:08.606 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:08.643 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:08.981 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:09.012 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:09.106 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:09.141 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:09.318 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:09.402 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:09.402 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:09.403 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:09.431 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:09.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:09.749 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:09.749 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:09.750 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:09.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:10.097 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:10.211 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:10.542 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:10.704 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:10.705 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-29T11:33:10.705 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:33:10.705 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-29T11:33:10.705 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:10.705 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:10.706 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-29T11:33:10.707 INFO:teuthology.orchestra.run.smithi033.stdout:Install 12 Packages 2024-03-29T11:33:10.707 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:10.707 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 37 M 2024-03-29T11:33:10.707 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 209 M 2024-03-29T11:33:10.707 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:10.972 INFO:teuthology.orchestra.run.smithi033.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 248 kB/s | 32 kB 00:00 2024-03-29T11:33:10.997 INFO:teuthology.orchestra.run.smithi033.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 687 kB/s | 17 kB 00:00 2024-03-29T11:33:11.022 INFO:teuthology.orchestra.run.smithi033.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-29T11:33:11.181 INFO:teuthology.orchestra.run.smithi033.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-29T11:33:11.306 INFO:teuthology.orchestra.run.smithi033.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 39 MB/s | 4.8 MB 00:00 2024-03-29T11:33:11.348 INFO:teuthology.orchestra.run.smithi033.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-29T11:33:11.432 INFO:teuthology.orchestra.run.smithi033.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-29T11:33:11.459 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:11.699 INFO:teuthology.orchestra.run.smithi033.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-29T11:33:12.469 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:12.505 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:12.540 INFO:teuthology.orchestra.run.smithi033.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 232 kB/s | 195 kB 00:00 2024-03-29T11:33:12.682 INFO:teuthology.orchestra.run.smithi033.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:33:13.116 INFO:teuthology.orchestra.run.smithi033.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 478 kB/s | 803 kB 00:01 2024-03-29T11:33:14.982 INFO:teuthology.orchestra.run.smithi033.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.8 MB/s | 7.4 MB 00:04 2024-03-29T11:33:14.986 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:14.986 INFO:teuthology.orchestra.run.smithi033.stdout:Total 8.7 MB/s | 37 MB 00:04 2024-03-29T11:33:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:15.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:15.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:15.030 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:15.057 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:15.057 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.434 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:15.439 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:15.439 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:15.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:16.163 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:16.239 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:16.270 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:16.369 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:16.407 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:16.443 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:16.594 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:16.605 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:16.607 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-03-29T11:33:16.608 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:16.609 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-03-29T11:33:16.609 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-29T11:33:16.609 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:16.631 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:16.714 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:17.088 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 18 kB 00:00 2024-03-29T11:33:17.113 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-03-29T11:33:17.155 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:33:17.180 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:33:17.206 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:33:17.231 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:33:17.256 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 180 kB/s | 52 kB 00:00 2024-03-29T11:33:17.403 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:17.498 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-29T11:33:17.521 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-29T11:33:17.766 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 398 kB/s | 203 kB 00:00 2024-03-29T11:33:17.866 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-29T11:33:18.116 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 256 kB/s | 90 kB 00:00 2024-03-29T11:33:18.183 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-29T11:33:18.186 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:18.186 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-03-29T11:33:18.187 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:18.301 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:18.302 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:18.830 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:18.835 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:18.997 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:19.050 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:19.114 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:19.151 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:19.193 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:19.286 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:19.395 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:19.444 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:19.910 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:19.938 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:20.163 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:20.207 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:20.241 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:20.289 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:20.311 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:21.139 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:21.139 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:21.139 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:21.139 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:21.139 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:21.140 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:21.504 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:21.551 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:21.551 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:21.551 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:21.552 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:21.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:21.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:21.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:21.553 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:21.553 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:21.553 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:21.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:22.334 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:22.456 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:22.457 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:22.486 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:22.486 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-29T11:33:22.486 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-29T11:33:22.487 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:22.488 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.8 M 2024-03-29T11:33:22.488 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-03-29T11:33:22.488 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:22.851 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-29T11:33:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:22.884 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:22.884 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:22.885 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:23.009 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:33:23.118 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 347 kB/s | 138 kB 00:00 2024-03-29T11:33:23.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:23.320 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 186 kB/s | 58 kB 00:00 2024-03-29T11:33:23.370 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-29T11:33:23.412 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 173 kB/s | 51 kB 00:00 2024-03-29T11:33:23.415 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:23.415 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2024-03-29T11:33:23.428 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:23.438 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:23.439 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:23.528 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:23.528 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:23.809 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:23.967 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:23.968 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-29T11:33:23.968 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:33:23.968 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-29T11:33:23.968 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:23.968 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:23.968 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:23.969 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-29T11:33:23.970 INFO:teuthology.orchestra.run.smithi033.stdout:Install 13 Packages 2024-03-29T11:33:23.970 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:23.970 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 2.0 M 2024-03-29T11:33:23.970 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 26 M 2024-03-29T11:33:23.970 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:24.245 INFO:teuthology.orchestra.run.smithi033.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 195 kB/s | 24 kB 00:00 2024-03-29T11:33:24.262 INFO:teuthology.orchestra.run.smithi033.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-03-29T11:33:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:33:24.337 INFO:teuthology.orchestra.run.smithi033.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:33:24.363 INFO:teuthology.orchestra.run.smithi033.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:33:24.388 INFO:teuthology.orchestra.run.smithi033.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-29T11:33:24.413 INFO:teuthology.orchestra.run.smithi033.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 178 kB/s | 52 kB 00:00 2024-03-29T11:33:24.547 INFO:teuthology.orchestra.run.smithi033.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-03-29T11:33:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:24.580 INFO:teuthology.orchestra.run.smithi033.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-29T11:33:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:24.672 INFO:teuthology.orchestra.run.smithi033.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 965 kB/s | 203 kB 00:00 2024-03-29T11:33:24.702 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:24.706 INFO:teuthology.orchestra.run.smithi033.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 6.5 MB/s | 1.0 MB 00:00 2024-03-29T11:33:24.726 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:24.731 INFO:teuthology.orchestra.run.smithi033.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-03-29T11:33:24.756 INFO:teuthology.orchestra.run.smithi033.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-29T11:33:24.761 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:24.761 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-29T11:33:24.761 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:24.775 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:24.775 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:24.872 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:24.873 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:25.528 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:25.529 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:25.529 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:25.529 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:25.529 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:25.529 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:25.529 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:25.719 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:25.776 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:25.837 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:25.839 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:25.875 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:25.926 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:26.025 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:26.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-03-29T11:33:26.140 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:26.194 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:26.644 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:26.794 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:26.795 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-29T11:33:26.796 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:26.796 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 964 k 2024-03-29T11:33:26.796 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.8 M 2024-03-29T11:33:26.796 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:26.922 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 627 kB/s | 79 kB 00:00 2024-03-29T11:33:26.958 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:27.007 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:27.090 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:27.119 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:27.514 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T11:33:27.515 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:27.516 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T11:33:27.516 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:27.536 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:27.536 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:27.610 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:27.610 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:27.789 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:27.924 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:27.997 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:28.324 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:28.324 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:28.324 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:28.324 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:28.324 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:28.324 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:28.324 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:28.572 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:28.690 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:28.691 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:29.019 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:29.019 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:29.019 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:29.019 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:29.019 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:29.019 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:29.161 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:29.319 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:29.320 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-29T11:33:29.320 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:33:29.320 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-29T11:33:29.320 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:29.320 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-03-29T11:33:29.321 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:29.322 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 6.8 M 2024-03-29T11:33:29.322 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 33 M 2024-03-29T11:33:29.322 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:29.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2024-03-29T11:33:29.913 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:30.069 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 332 k 2024-03-29T11:33:30.071 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.5 M 2024-03-29T11:33:30.071 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:30.151 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 545 kB/s | 138 kB 00:00 2024-03-29T11:33:30.176 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 891 kB/s | 248 kB 00:00 2024-03-29T11:33:30.210 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 495 kB/s | 69 kB 00:00 2024-03-29T11:33:30.443 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 216 kB/s | 58 kB 00:00 2024-03-29T11:33:30.535 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 560 kB/s | 51 kB 00:00 2024-03-29T11:33:30.585 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 513 kB/s | 264 kB 00:00 2024-03-29T11:33:30.586 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:30.587 INFO:teuthology.orchestra.run.smithi153.stdout:Total 644 kB/s | 332 kB 00:00 2024-03-29T11:33:30.587 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:30.595 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:30.595 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:30.602 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T11:33:30.638 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:30.638 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:30.790 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:30.836 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.2 MB/s | 5.8 MB 00:00 2024-03-29T11:33:30.837 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:30.837 INFO:teuthology.orchestra.run.smithi033.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-29T11:33:30.851 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:30.863 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:30.864 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:30.959 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:30.959 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:30.970 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:31.004 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:31.020 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:31.020 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:31.020 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:31.020 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:31.260 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:31.433 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:31.488 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:31.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:32.043 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:32.111 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:32.134 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:32.202 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:32.232 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:32.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-03-29T11:33:33.019 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:33.024 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:33.024 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:33.024 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:33.024 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:33.024 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:33.025 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:33.171 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-29T11:33:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 127 k 2024-03-29T11:33:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 456 k 2024-03-29T11:33:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:33.336 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:33.337 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:33.365 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 656 kB/s | 127 kB 00:00 2024-03-29T11:33:33.366 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:33.366 INFO:teuthology.orchestra.run.smithi153.stdout:Total 652 kB/s | 127 kB 00:00 2024-03-29T11:33:33.366 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:33.372 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:33.372 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:33.427 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:33.428 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:33.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-fuse 2024-03-29T11:33:33.576 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:33.690 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.137 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:34.157 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.294 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-03-29T11:33:34.295 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 964 k 2024-03-29T11:33:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 2.8 M 2024-03-29T11:33:34.296 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:34.444 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:34.444 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:34.444 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:34.445 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 530 kB/s | 79 kB 00:00 2024-03-29T11:33:34.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-03-29T11:33:34.828 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 885 kB 00:00 2024-03-29T11:33:34.830 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:34.830 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.8 MB/s | 964 kB 00:00 2024-03-29T11:33:34.830 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:34.852 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:34.852 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:34.929 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:34.929 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:35.116 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:35.250 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:35.267 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:35.340 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:35.350 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:35.404 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:35.405 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:33:35.405 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:35.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:36.044 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:36.195 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 31 k 2024-03-29T11:33:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 140 k 2024-03-29T11:33:36.197 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:36.427 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 135 kB/s | 31 kB 00:00 2024-03-29T11:33:36.428 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:36.428 INFO:teuthology.orchestra.run.smithi153.stdout:Total 134 kB/s | 31 kB 00:00 2024-03-29T11:33:36.428 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:36.431 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:36.431 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:36.444 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:36.444 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:36.461 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:36.462 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:36.581 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:36.676 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:36.711 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-volume 2024-03-29T11:33:37.089 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:37.292 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:37.359 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:37.359 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:37.359 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:37.360 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:37.360 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:37.360 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:37.451 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:37.451 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:37.451 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-29T11:33:37.451 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:37.451 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-03-29T11:33:37.452 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:37.453 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 332 k 2024-03-29T11:33:37.453 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.5 M 2024-03-29T11:33:37.453 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:37.525 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-03-29T11:33:37.584 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 523 kB/s | 69 kB 00:00 2024-03-29T11:33:37.743 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 910 kB/s | 264 kB 00:00 2024-03-29T11:33:37.744 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:37.744 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-29T11:33:37.744 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:37.753 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:37.753 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:37.790 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:37.790 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:37.940 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:38.093 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:38.132 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:38.170 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:38.186 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:38.187 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:38.187 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:38.187 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:38.192 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:38.244 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:38.245 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:33:38.245 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:38.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-03-29T11:33:38.862 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:38.929 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:38.959 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:39.011 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:39.012 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:33:39.012 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:39.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:39.586 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados-devel 2024-03-29T11:33:39.629 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:39 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:39.728 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:39.780 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:39.781 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:33:39.781 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:39.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-03-29T11:33:40.167 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:40.324 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:40.324 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 127 k 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 456 k 2024-03-29T11:33:40.325 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:40.402 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:40.502 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:40.554 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:40.555 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:33:40.555 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:40.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-03-29T11:33:40.765 INFO:teuthology.orchestra.run.smithi033.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 288 kB/s | 127 kB 00:00 2024-03-29T11:33:40.766 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:40.766 INFO:teuthology.orchestra.run.smithi033.stdout:Total 287 kB/s | 127 kB 00:00 2024-03-29T11:33:40.766 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:40.772 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:40.772 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:40.831 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:40.832 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:40.987 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:41.109 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:41.170 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:41.269 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:41.320 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:41.321 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:33:41.321 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:41.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-03-29T11:33:41.529 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:41.833 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:41.833 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:41.833 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:41.833 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:41.833 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:41.834 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:41.939 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:42.039 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:42.045 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs2 2024-03-29T11:33:42.091 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:42.092 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-29T11:33:42.092 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:42.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-03-29T11:33:42.627 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:34 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:42.706 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:42 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:42.729 INFO:teuthology.orchestra.run.smithi033.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:42.786 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:42.787 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:42.787 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:42.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:42.854 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:42.854 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-29T11:33:42.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:42.856 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 85 k 2024-03-29T11:33:42.856 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 231 k 2024-03-29T11:33:42.857 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:43.197 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 250 kB/s | 85 kB 00:00 2024-03-29T11:33:43.198 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:43.198 INFO:teuthology.orchestra.run.smithi153.stdout:Total 249 kB/s | 85 kB 00:00 2024-03-29T11:33:43.198 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:43.205 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:43.205 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:43.260 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:43.260 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:43.395 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:43.426 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:43.495 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:43.584 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:43.585 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:43.585 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-29T11:33:43.585 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:43.585 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:43.585 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 31 k 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 140 k 2024-03-29T11:33:43.586 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:43.743 INFO:teuthology.orchestra.run.smithi033.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 199 kB/s | 31 kB 00:00 2024-03-29T11:33:43.744 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:43.744 INFO:teuthology.orchestra.run.smithi033.stdout:Total 197 kB/s | 31 kB 00:00 2024-03-29T11:33:43.744 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:43.747 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:43.748 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:43.777 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:43.777 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:43.896 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:43.927 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:43.994 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:44.219 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:44.219 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:44.219 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:44.219 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:44.219 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:44.219 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:44.354 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:44.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-03-29T11:33:44.617 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:44.617 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:44.617 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:44.617 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:44.617 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:44.617 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:44.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados2 2024-03-29T11:33:45.036 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:45.195 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:45.195 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:45.195 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-29T11:33:45.195 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.1 M 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 13 M 2024-03-29T11:33:45.196 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:45.342 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:45.442 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:45.497 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:45.498 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:45.498 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:45.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librbd1 2024-03-29T11:33:46.158 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:46.257 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:46.312 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:46.313 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:46.354 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rados 2024-03-29T11:33:46.407 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.5 MB/s | 3.1 MB 00:01 2024-03-29T11:33:46.408 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:46.409 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.5 MB/s | 3.1 MB 00:01 2024-03-29T11:33:46.409 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:46.420 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:46.420 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:46.496 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:46.497 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:46.931 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:46.970 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:47.031 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:47.033 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:47.057 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:47.058 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:47.058 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:33:47.058 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:47.058 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:47.058 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:47.086 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:47.087 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:47.087 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:47.138 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rgw 2024-03-29T11:33:47.717 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:39 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:47.819 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:47.875 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:47.876 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:47.876 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:47.925 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-cephfs 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:48.507 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:48.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-03-29T11:33:48.609 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:48.664 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:48.665 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:48.665 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:48.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rbd 2024-03-29T11:33:49.122 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:49 ago on Fri 29 Mar 2024 11:31:00 AM UTC. 2024-03-29T11:33:49.273 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-29T11:33:49.273 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:49.273 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 171 k 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 494 k 2024-03-29T11:33:49.274 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-29T11:33:49.315 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:49.418 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:49.473 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:49.474 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-29T11:33:49.474 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:49.486 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 807 kB/s | 171 kB 00:00 2024-03-29T11:33:49.487 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:49.487 INFO:teuthology.orchestra.run.smithi153.stdout:Total 802 kB/s | 171 kB 00:00 2024-03-29T11:33:49.488 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-29T11:33:49.495 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-29T11:33:49.495 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-29T11:33:49.519 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-fuse 2024-03-29T11:33:49.551 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-29T11:33:49.552 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-29T11:33:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-29T11:33:49.810 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:50.093 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:42 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:50.248 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:50.249 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:50.249 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:50.249 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:50.249 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:50.249 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:50.249 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:33:50.250 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:50.250 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:50.250 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:50.250 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-29T11:33:50.250 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:50.250 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 85 k 2024-03-29T11:33:50.251 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 231 k 2024-03-29T11:33:50.251 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:50.463 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 401 kB/s | 85 kB 00:00 2024-03-29T11:33:50.464 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:50.464 INFO:teuthology.orchestra.run.smithi033.stdout:Total 399 kB/s | 85 kB 00:00 2024-03-29T11:33:50.464 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:50.471 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:50.471 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:50.523 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:50.523 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:50.523 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-29T11:33:50.524 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:50.524 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:33:50.524 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-29T11:33:50.529 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:50.529 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:50.672 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:50.765 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:50.766 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:51.134 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:51.404 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:51.404 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:51.405 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:51.405 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:51.405 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:51.405 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:51.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-mirror 2024-03-29T11:33:52.178 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:44 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:52.334 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:52.335 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:52.335 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-29T11:33:52.335 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:52.335 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 3.1 M 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 13 M 2024-03-29T11:33:52.336 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:53.157 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.8 MB/s | 3.1 MB 00:00 2024-03-29T11:33:53.158 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:53.158 INFO:teuthology.orchestra.run.smithi033.stdout:Total 3.8 MB/s | 3.1 MB 00:00 2024-03-29T11:33:53.158 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:53.170 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:53.170 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:53.250 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:53.250 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:53.707 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:53.758 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:53.782 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:53.782 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:53.782 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:33:53.782 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:53.782 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:53.783 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:54.865 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:54.865 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:54.865 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:54.865 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:54.865 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:54.865 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:55.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-nbd 2024-03-29T11:33:55.681 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:47 ago on Fri 29 Mar 2024 11:31:08 AM UTC. 2024-03-29T11:33:55.835 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-29T11:33:55.836 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:55.836 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:55.836 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:55.836 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-29T11:33:55.836 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 171 k 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 494 k 2024-03-29T11:33:55.837 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-29T11:33:56.060 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 767 kB/s | 171 kB 00:00 2024-03-29T11:33:56.061 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:56.061 INFO:teuthology.orchestra.run.smithi033.stdout:Total 763 kB/s | 171 kB 00:00 2024-03-29T11:33:56.062 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-29T11:33:56.070 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-29T11:33:56.070 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-29T11:33:56.129 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-29T11:33:56.129 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-29T11:33:56.284 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-29T11:33:56.387 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:56.781 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:57.046 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:57.046 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:57.046 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-29T11:33:57.047 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:57.047 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:33:57.047 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-29T11:33:57.318 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:57.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:33:57.459 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:33:57.480 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:57.481 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:57.481 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:33:57.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:33:57.696 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:33:57.720 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:57.721 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:57.721 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:33:57.723 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:33:57.723 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:33:57.723 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:33:57.751 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:33:57.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:33:57.792 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:33:57.793 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:33:57.793 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:33:57.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:33:57.886 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:33:57.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:33:57.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:33:57.985 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:33:57.985 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:33:57.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:33:58.012 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:33:58.081 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:33:58.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:33:58.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:33:58.177 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:33:58.177 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:33:58.177 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:33:58.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:33:58.270 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:33:58.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:33:58.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:33:58.365 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:33:58.455 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:33:58.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2024-03-29T11:33:58.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-03-29T11:33:58.485 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:33:58.485 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:33:58.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:33:58.551 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:33:58.551 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:33:58.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:33:58.630 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:33:58.630 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:33:58.630 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622711', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLfB6oBFzuCD9lJafl5tdfIzT0/P1jWSao7uthDO+9x9yuQW/zyI15ZyOrti/J982iJuh3X09GkUwuH85bqbgg=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUFjPwIOVgPGNqn7AKOiSDnNg4zyeWjB81qwLpb6ObHbse3OpFmOyuwE9sNtnpAqwNhxHi5RyHAklknpOIBC7s='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:33:58.630 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:33:58.657 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:33:58.675 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:33:58.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:33:58.741 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:33:58.742 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:33:58.791 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:33:58.791 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-03-29T11:33:58.845 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:33:58.845 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:58.845 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:33:58.845 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:58.845 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:58.846 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-29 11:33:54.032325985 +0000 2024-03-29T11:33:58.846 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-29 11:30:08.755757164 +0000 2024-03-29T11:33:58.846 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-29 11:30:08.755757164 +0000 2024-03-29T11:33:58.846 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-29 11:30:08.755757164 +0000 2024-03-29T11:33:58.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:33:58.910 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-29T11:33:58.910 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-29T11:33:58.910 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000196516 s, 2.6 MB/s 2024-03-29T11:33:58.911 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:33:58.967 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-29 11:33:54.032325985 +0000 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-29 11:30:09.105751063 +0000 2024-03-29T11:33:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-29 11:30:09.105751063 +0000 2024-03-29T11:33:59.022 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-29 11:30:09.105751063 +0000 2024-03-29T11:33:59.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:33:59.086 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-29T11:33:59.086 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-29T11:33:59.086 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000173244 s, 3.0 MB/s 2024-03-29T11:33:59.087 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:33:59.142 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-03-29T11:33:59.196 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:33:59.196 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:59.196 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:33:59.196 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:59.196 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:59.197 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-29 11:33:54.033325968 +0000 2024-03-29T11:33:59.197 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-29 11:30:09.454744979 +0000 2024-03-29T11:33:59.197 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-29 11:30:09.454744979 +0000 2024-03-29T11:33:59.197 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-29 11:30:09.454744979 +0000 2024-03-29T11:33:59.197 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:33:59.261 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-29T11:33:59.261 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-29T11:33:59.261 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000193246 s, 2.6 MB/s 2024-03-29T11:33:59.262 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:33:59.317 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-29 11:33:54.033325968 +0000 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-29 11:30:09.788739157 +0000 2024-03-29T11:33:59.371 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-29 11:30:09.788739157 +0000 2024-03-29T11:33:59.372 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-29 11:30:09.788739157 +0000 2024-03-29T11:33:59.372 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:33:59.436 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-29T11:33:59.436 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-29T11:33:59.437 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000193923 s, 2.6 MB/s 2024-03-29T11:33:59.438 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:33:59.493 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:59.493 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:33:59.493 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:33:59.509 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:33:59.509 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-29T11:33:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:33:59.566 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:59.566 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:33:59.566 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:59.566 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:59.566 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-29 11:33:47.318481838 +0000 2024-03-29T11:33:59.567 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-29 11:30:14.429751662 +0000 2024-03-29T11:33:59.567 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-29 11:30:14.429751662 +0000 2024-03-29T11:33:59.567 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-29 11:30:14.429751662 +0000 2024-03-29T11:33:59.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:33:59.633 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-29T11:33:59.634 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-29T11:33:59.634 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000266428 s, 1.9 MB/s 2024-03-29T11:33:59.635 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:33:59.691 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-29T11:33:59.747 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-29 11:33:47.318481838 +0000 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-29 11:30:14.781745363 +0000 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-29 11:30:14.781745363 +0000 2024-03-29T11:33:59.748 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-29 11:30:14.781745363 +0000 2024-03-29T11:33:59.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:33:59.814 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-29T11:33:59.814 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-29T11:33:59.814 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000191116 s, 2.7 MB/s 2024-03-29T11:33:59.815 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:33:59.871 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-29 11:33:47.319481820 +0000 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-29 11:30:15.116739368 +0000 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-29 11:30:15.116739368 +0000 2024-03-29T11:33:59.926 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-29 11:30:15.116739368 +0000 2024-03-29T11:33:59.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:33:59.991 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-29T11:33:59.992 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-29T11:33:59.992 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000210053 s, 2.4 MB/s 2024-03-29T11:33:59.993 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:34:00.049 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-29 11:33:47.319481820 +0000 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-29 11:30:15.459733230 +0000 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-29 11:30:15.459733230 +0000 2024-03-29T11:34:00.104 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-29 11:30:15.459733230 +0000 2024-03-29T11:34:00.105 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:34:00.169 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-29T11:34:00.170 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-29T11:34:00.170 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00021111 s, 2.4 MB/s 2024-03-29T11:34:00.171 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:34:00.227 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:00.227 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:00.227 INFO:tasks.ceph:Generating config... 2024-03-29T11:34:00.228 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:34:00.228 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:34:00.228 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:34:00.228 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:34:00.228 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:34:00.228 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:34:00.228 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:34:00.228 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:34:00.228 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:34:00.229 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:34:00.229 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:34:00.229 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:34:00.229 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:34:00.229 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:34:00.230 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:34:00.231 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:34:00.231 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:34:00.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:00.290 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:34:00.293 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:34:00.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:00.366 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.33:6789]'), ('mon.c', '[v1:172.21.15.33:6790]'), ('mon.b', '[v1:172.21.15.153:6789]')] 2024-03-29T11:34:00.367 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.33:6789],[v1:172.21.15.33:6790],[v1:172.21.15.153:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:34:00.367 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:00.368 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:00.421 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.33:6789]' --addv c '[v1:172.21.15.33:6790]' --addv b '[v1:172.21.15.153:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:00.506 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: generated fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = quincy 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 0 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:min_mon_release 17 (quincy) 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:election_strategy: 1 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:00.507 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:34:00.508 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:00.523 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6... 2024-03-29T11:34:00.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:00.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:00.614 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-03-29T11:34:00.614 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2024-03-29T11:34:00.614 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:00.614 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2024-03-29T11:34:00.614 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.614 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2024-03-29T11:34:00.614 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2024-03-29T11:34:00.615 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.616 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = "[v1:172.21.15.33:6789],[v1:172.21.15.33:6790],[v1:172.21.15.153:6789]" 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: lockdep = True 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: mon election default strategy = 1 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: ms bind msgr2 = False 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 5000 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: ms type = async 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.617 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2024-03-29T11:34:00.618 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2024-03-29T11:34:00.619 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore zero block detection = True 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 20 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 20 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 10 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:00.620 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd class default list = * 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd class load list = * 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd client watch timeout = 120 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify missing on start = True 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue = debug_random 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2024-03-29T11:34:00.621 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2024-03-29T11:34:00.622 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:00.623 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon scrub interval = 300 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on pool no app = False 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: paxos service trim min = 10 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2024-03-29T11:34:00.624 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: debug objecter = 20 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout: debug rados = 20 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.c] 2024-03-29T11:34:00.625 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.b] 2024-03-29T11:34:00.626 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-03-29T11:34:00.626 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-03-29T11:34:00.626 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:00.626 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-03-29T11:34:00.626 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.627 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:00.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "[v1:172.21.15.33:6789],[v1:172.21.15.33:6790],[v1:172.21.15.153:6789]" 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: lockdep = True 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2024-03-29T11:34:00.629 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 1 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr2 = False 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 5000 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-03-29T11:34:00.630 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-03-29T11:34:00.631 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.632 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:00.633 INFO:teuthology.orchestra.run.smithi153.stdout: osd class default list = * 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd class load list = * 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd client watch timeout = 120 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-03-29T11:34:00.634 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-03-29T11:34:00.635 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on pool no app = False 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 10 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-03-29T11:34:00.637 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout: debug objecter = 20 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout: debug rados = 20 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-03-29T11:34:00.638 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-03-29T11:34:00.639 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:34:00.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:34:00.695 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:34:00.695 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:00.695 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:34:00.709 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:00.709 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:34:00.763 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:34:00.763 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:00.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:00.764 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:00.843 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:00.843 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:00.897 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:34:00.898 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:00.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:00.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:00.937 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:00.938 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:00.993 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:34:00.993 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:34:00.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:01.047 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:01.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:01.108 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:01.111 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:34:01.111 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:34:01.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:01.153 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:01.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:01.210 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:01.226 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:34:01.226 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:01.226 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:01.253 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:01.253 INFO:tasks.ceph:role: osd.0 2024-03-29T11:34:01.253 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:34:01.253 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:01.326 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:01.327 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:01.824 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-29T11:34:01.826 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-29T11:34:01.826 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:01.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:02.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:02.083 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:02.083 INFO:tasks.ceph:role: osd.1 2024-03-29T11:34:02.083 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:34:02.083 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:02.157 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:02.158 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:02.599 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-29T11:34:02.601 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-29T11:34:02.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:02.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:02.722 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:02.789 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:02.789 INFO:tasks.ceph:role: osd.2 2024-03-29T11:34:02.789 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:34:02.789 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:02.863 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:02.864 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:02.864 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:03.364 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-29T11:34:03.366 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-29T11:34:03.366 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:03.498 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:03.530 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:03.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:03.597 INFO:tasks.ceph:role: osd.3 2024-03-29T11:34:03.598 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:34:03.598 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:03.672 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:04.167 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-29T11:34:04.169 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-29T11:34:04.169 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:04.313 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:04.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:04.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:04.450+0000 7feede404640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:34:04.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:04.451+0000 7feede404640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:34:04.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:04.451+0000 7feede404640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:04.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:04.451+0000 7feede404640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:04.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:04.451+0000 7feede404640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:04.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:04.451+0000 7feede404640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:34:07.298 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:07.329 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:07.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:07.424+0000 7fda55aa4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:34:07.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:07.424+0000 7fda55aa4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:34:07.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:07.424+0000 7fda55aa4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:07.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:07.424+0000 7fda55aa4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:07.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:07.424+0000 7fda55aa4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:07.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:07.425+0000 7fda55aa4640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:34:10.273 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:10.304 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:10.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:10.399+0000 7fa3e1a18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:34:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:10.399+0000 7fa3e1a18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:34:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:10.399+0000 7fa3e1a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:10.399+0000 7fa3e1a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:10.399+0000 7fa3e1a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:10.399+0000 7fa3e1a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:34:13.255 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:13.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:13.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:13.385+0000 7f913a80f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:34:13.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:13.385+0000 7f913a80f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:34:13.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:13.385+0000 7f913a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:13.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:13.385+0000 7f913a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:13.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:13.385+0000 7f913a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:13.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:13.385+0000 7f913a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:34:16.238 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:16.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:16.302 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:16.302 INFO:tasks.ceph:role: osd.4 2024-03-29T11:34:16.302 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:34:16.302 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:16.380 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:16.900 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-29T11:34:16.902 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-29T11:34:16.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:17.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:17.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:17.139 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:17.139 INFO:tasks.ceph:role: osd.5 2024-03-29T11:34:17.139 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:34:17.139 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:17.215 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:17.216 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:17.216 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:17.216 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:17.216 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:17.707 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-29T11:34:17.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-29T11:34:17.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:17.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:17.827 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:17.895 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:17.895 INFO:tasks.ceph:role: osd.6 2024-03-29T11:34:17.895 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:34:17.895 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:17.969 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:17.969 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:17.969 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:17.970 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:17.970 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:17.970 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:17.970 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:17.970 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:17.970 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:17.970 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:18.488 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-29T11:34:18.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-29T11:34:18.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:18.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:18.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:18.723 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:18.723 INFO:tasks.ceph:role: osd.7 2024-03-29T11:34:18.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:34:18.724 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:18.797 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:18.797 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:18.798 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:19.288 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-29T11:34:19.290 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-29T11:34:19.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:19.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:19.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:19.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:19.543+0000 7f0899993640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:34:19.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:19.543+0000 7f0899993640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:34:19.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:19.544+0000 7f0899993640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:19.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:19.544+0000 7f0899993640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:19.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:19.544+0000 7f0899993640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:19.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:19.544+0000 7f0899993640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:34:22.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:22.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:22.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:22.516+0000 7f1483945640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:34:22.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:22.516+0000 7f1483945640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:34:22.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:22.516+0000 7f1483945640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:22.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:22.516+0000 7f1483945640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:22.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:22.516+0000 7f1483945640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:22.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:22.517+0000 7f1483945640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:34:25.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:25.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:25.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:25.504+0000 7f4531d8f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:34:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:25.504+0000 7f4531d8f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:34:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:25.504+0000 7f4531d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:25.504+0000 7f4531d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:25.504+0000 7f4531d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:25.505+0000 7f4531d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:34:28.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:28.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:28.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:28.473+0000 7fb28e619640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:34:28.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:28.473+0000 7fb28e619640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:34:28.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:28.473+0000 7fb28e619640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:28.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:28.473+0000 7fb28e619640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:28.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:28.473+0000 7fb28e619640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:28.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:28.474+0000 7fb28e619640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:34:31.320 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:31.351 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:34:31.352 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:31.352 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:34:31.384 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:31.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:34:31.454 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:31.454 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:34:31.524 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:31.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:34:31.593 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:31.594 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:34:31.662 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:31.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:34:31.694 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:31.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:34:31.766 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:31.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:34:31.837 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:31.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:34:31.908 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:31.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:34:31.976 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:31.977 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:34:31.993 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:31.993 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:34:32.033 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:34:32.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:32.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:32.075 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.y] 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAppwZmrPsaAxAAUVztiKCO9JDiJTT6G0qiow== 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAspwZmOv3nGhAAfRHble7UKJgsrj6TWcJ0KQ== 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAvpwZmKPZUGRAASRuivIR+9Rgwc1dxZ5MJnw== 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAypwZmJfnRFxAAka/FXGrn+ZDLo2bjV+rCUg== 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.3] 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA1pwZm0CwNFxAAVlzhrn0wr6HXiBxYLmOzdQ== 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAppwZmhU/ABhAAyT/SRLIPvYezVPNitkZJRA== 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.4] 2024-03-29T11:34:32.076 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA7pwZmIPZ/IBAAZ6i0cEnkF/+kPkX5uVfe7A== 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.5] 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA+pwZmVlLdHhAAEPGgQwRhfaQXxQ775HTA7Q== 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.6] 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBBpwZmr6kkHhAA8vkr8X5P06OC9l3nV8k4MQ== 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.7] 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBEpwZmYZNQHBAAxB+SWlJ0tw/tRq2b4zw40w== 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAppwZmLydrCRAAv/hLXEc1cjgOW6PmIaQEFg== 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout:[client.1] 2024-03-29T11:34:32.077 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAppwZmAb/UDBAA08A4KjM1QFPDAT8rQJif5g== 2024-03-29T11:34:32.099 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.y] 2024-03-29T11:34:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAppwZmrPsaAxAAUVztiKCO9JDiJTT6G0qiow== 2024-03-29T11:34:32.099 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-03-29T11:34:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAspwZmOv3nGhAAfRHble7UKJgsrj6TWcJ0KQ== 2024-03-29T11:34:32.099 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-03-29T11:34:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAvpwZmKPZUGRAASRuivIR+9Rgwc1dxZ5MJnw== 2024-03-29T11:34:32.099 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAypwZmJfnRFxAAka/FXGrn+ZDLo2bjV+rCUg== 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA1pwZm0CwNFxAAVlzhrn0wr6HXiBxYLmOzdQ== 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAppwZmhU/ABhAAyT/SRLIPvYezVPNitkZJRA== 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA7pwZmIPZ/IBAAZ6i0cEnkF/+kPkX5uVfe7A== 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA+pwZmVlLdHhAAEPGgQwRhfaQXxQ775HTA7Q== 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.6] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBBpwZmr6kkHhAA8vkr8X5P06OC9l3nV8k4MQ== 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.7] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBEpwZmYZNQHBAAxB+SWlJ0tw/tRq2b4zw40w== 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-29T11:34:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAppwZmLydrCRAAv/hLXEc1cjgOW6PmIaQEFg== 2024-03-29T11:34:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-03-29T11:34:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQAppwZmAb/UDBAA08A4KjM1QFPDAT8rQJif5g== 2024-03-29T11:34:32.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:32.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:32.195 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.249 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.302 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.358 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:32.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:32.466 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.468 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.520 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:32.681 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:32.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:32.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:32.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:32.794 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:34:32.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:32.820 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:32.916 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:32.942 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:33.009 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:33.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:33.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:33.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:33.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:33.284 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:33.286 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:33.339 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:34:33.339 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:34:33.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:34:33.342 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:34:33.342 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:34:33.342 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:34:33.344 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:34:33.344 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:34:33.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:34:33.381 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:34:33.381 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:34:33.381 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:34:33.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:34:33.384 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:34:33.384 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:34:33.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:34:33.386 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:34:33.386 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:34:33.386 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.490+0000 7f9083738640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.490+0000 7f9083738640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.490+0000 7f9083738640 10 monclient: monmap: 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:33.491903+0000 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:33.491903+0000 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:33.488 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:33.489 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:33.489 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:33.489 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.490+0000 7f9083738640 1 Processor -- start 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 1 -- start start 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient: init 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient(hunting): picked mon.noname-c con 0x7f907c0b8240 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient(hunting): picked mon.noname-a con 0x7f907c0b1f60 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient(hunting): picked mon.noname-b con 0x7f907c09c8c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:33.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c09bd80 con 0x7f907c0b1f60 2024-03-29T11:34:33.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c09b960 con 0x7f907c0b8240 2024-03-29T11:34:33.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c0b2ec0 con 0x7f907c09c8c0 2024-03-29T11:34:33.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9083738640 10 monclient(hunting): authenticate will time out at 829.756374s 2024-03-29T11:34:33.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9081cd5640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f907c09c8c0 0x7f907c14d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:46428/0 (socket says 172.21.15.33:46428) 2024-03-29T11:34:33.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:33.491+0000 7f9081cd5640 1 -- 172.21.15.33:0/3448530237 learned_addr learned my addr 172.21.15.33:0/3448530237 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient(hunting): tick 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 1 monclient(hunting): continuing hunt 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 1 -- 172.21.15.33:0/3448530237 >> v1:172.21.15.33:6790/0 conn(0x7f907c09c8c0 legacy=0x7f907c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 1 -- 172.21.15.33:0/3448530237 >> v1:172.21.15.153:6789/0 conn(0x7f907c0b8240 legacy=0x7f907c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 1 -- 172.21.15.33:0/3448530237 >> v1:172.21.15.33:6789/0 conn(0x7f907c0b1f60 legacy=0x7f907c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:36.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f90600036e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f90600072d0 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f906000b000 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 1 -- 172.21.15.33:0/3448530237 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c13fb60 con 0x7f906000b000 2024-03-29T11:34:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 1 -- 172.21.15.33:0/3448530237 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906000fc90 con 0x7f90600072d0 2024-03-29T11:34:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 1 -- 172.21.15.33:0/3448530237 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9060010ef0 con 0x7f90600036e0 2024-03-29T11:34:36.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:36.492+0000 7f9072ffd640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:38.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.663+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172713593 0 0) 0x7f907c13fb60 con 0x7f906000b000 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.663+0000 7f9073fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.663+0000 7f9073fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.663+0000 7f9073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.663+0000 7f9073fff640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.663+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9068003520 con 0x7f906000b000 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115305995 0 0) 0x7f9060010ef0 con 0x7f90600036e0 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f907c13fb60 con 0x7f90600036e0 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 935234632 0 0) 0x7f907c13fb60 con 0x7f90600036e0 2024-03-29T11:34:38.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 >> v1:172.21.15.153:6789/0 conn(0x7f90600072d0 legacy=0x7f9060009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 >> v1:172.21.15.33:6789/0 conn(0x7f906000b000 legacy=0x7f906000d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 monclient: found mon.noname-b 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9060012150 con 0x7f90600036e0 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9078001610 con 0x7f90600036e0 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 10 monclient: dump: 2024-03-29T11:34:38.667 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9083738640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9083738640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.664+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9078001770 con 0x7f90600036e0 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.665+0000 7f9073fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.665+0000 7f9073fff640 1 -- 172.21.15.33:0/3448530237 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9078001b80 con 0x7f90600036e0 2024-03-29T11:34:38.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.665+0000 7f9073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.665+0000 7f9073fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.665+0000 7f9073fff640 10 monclient: dump: 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.665+0000 7f9083738640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:38.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.665+0000 7f9083738640 4 set_mon_vals no callback set 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.666+0000 7f9083738640 10 monclient: shutdown 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.666+0000 7f9083738640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.666+0000 7f9083738640 1 -- 172.21.15.33:0/3448530237 >> v1:172.21.15.33:6790/0 conn(0x7f90600036e0 legacy=0x7f9060005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.666+0000 7f9083738640 1 -- 172.21.15.33:0/3448530237 shutdown_connections 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.666+0000 7f9083738640 1 -- 172.21.15.33:0/3448530237 >> 172.21.15.33:0/3448530237 conn(0x7f907c066b00 msgr2=0x7f907c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.666+0000 7f9083738640 1 -- 172.21.15.33:0/3448530237 shutdown_connections 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.666+0000 7f9083738640 1 -- 172.21.15.33:0/3448530237 wait complete. 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.667+0000 7f9083738640 10 monclient: build_initial_monmap 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.667+0000 7f9083738640 10 monclient: monmap: 2024-03-29T11:34:38.676 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.667+0000 7f9083738640 1 librados: starting msgr at 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.667+0000 7f9083738640 1 librados: starting objecter 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.668+0000 7f9083738640 1 Processor -- start 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.668+0000 7f9083738640 1 -- start start 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.668+0000 7f9083738640 1 librados: setting wanted keys 2024-03-29T11:34:38.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.668+0000 7f9083738640 1 librados: calling monclient init 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.668+0000 7f9083738640 10 monclient: init 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 10 monclient(hunting): picked mon.noname-c con 0x7f90600036e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f90814d4640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f90600036e0 0x7f907c1dac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:45790/0 (socket says 172.21.15.33:45790) 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f90814d4640 1 -- 172.21.15.33:0/1060039726 learned_addr learned my addr 172.21.15.33:0/1060039726 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 10 monclient(hunting): picked mon.noname-a con 0x7f90600072d0 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9071ffb640 10 client.?.objecter ms_handle_connect 0x7f90600036e0 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9071ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 10 monclient(hunting): picked mon.noname-b con 0x7f906000b000 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c1da760 con 0x7f90600072d0 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9071ffb640 10 client.?.objecter ms_handle_connect 0x7f90600072d0 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9071ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:38.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c1db390 con 0x7f906000b000 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c1db520 con 0x7f90600036e0 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f90814d4640 1 -- 172.21.15.33:0/1060039726 >> v1:172.21.15.33:6790/0 conn(0x7f90600036e0 legacy=0x7f907c1dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f90814d4640 1 -- 172.21.15.33:0/1060039726 >> v1:172.21.15.33:6790/0 conn(0x7f90600036e0 legacy=0x7f907c1dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f90814d4640 1 --1- 172.21.15.33:0/1060039726 >> v1:172.21.15.33:6790/0 conn(0x7f90600036e0 0x7f907c1dac80 :-1 s=OPENED pgs=19 cs=1 l=1).handle_message read tag failed 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f90814d4640 1 --1- 172.21.15.33:0/1060039726 >> v1:172.21.15.33:6790/0 conn(0x7f90600036e0 0x7f907c1dac80 :-1 s=OPENED pgs=19 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9071ffb640 10 client.?.objecter ms_handle_connect 0x7f906000b000 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.669+0000 7f9071ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9083738640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9083738640 10 monclient(hunting): authenticate will time out at 834.934682s 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9071ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.33:6790/0 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5467342 0 0) 0x7f907c1da760 con 0x7f90600072d0 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9071ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9071ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9071ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.670+0000 7f9071ffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:34:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90600106c0 con 0x7f90600072d0 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1003608549 0 0) 0x7f90600106c0 con 0x7f90600072d0 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f907c1da760 con 0x7f90600072d0 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f906c0017a0 con 0x7f90600072d0 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3715835688 0 0) 0x7f907c1da760 con 0x7f90600072d0 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 >> v1:172.21.15.33:6790/0 conn(0x7f90600036e0 legacy=0x7f907c1dac80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.671+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 >> v1:172.21.15.153:6789/0 conn(0x7f906000b000 legacy=0x7f907c141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 1 monclient: found mon.a 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c140f80 con 0x7f90600072d0 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 10 monclient: _renew_subs 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c1cbdd0 con 0x7f90600072d0 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f906c001d10 con 0x7f90600072d0 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f906c003790 con 0x7f90600072d0 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 monclient: dump: 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f90737fe640 4 set_mon_vals no callback set 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f906c003d00 con 0x7f90600072d0 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 10 monclient: _renew_subs 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c13a7a0 con 0x7f90600072d0 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9083738640 1 librados: init done 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1166340729 0 0) 0x7f906c004640 con 0x7f90600072d0 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 10 client.4113.objecter ms_dispatch 0x7f907c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:34:38.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.672+0000 7f9071ffb640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:34:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.673+0000 7f9083738640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.673+0000 7f9083738640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.673+0000 7f9083738640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.673+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c09b170 con 0x7f90600072d0 2024-03-29T11:34:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.677+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f906c004800 con 0x7f90600072d0 2024-03-29T11:34:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.677+0000 7f9071ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:38.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.677+0000 7f9071ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:38.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.747+0000 7f9083738640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:38.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.747+0000 7f9083738640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:38.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.747+0000 7f9083738640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:38.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.747+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f907c141170 con 0x7f90600072d0 2024-03-29T11:34:38.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.749+0000 7f9071ffb640 1 -- 172.21.15.33:0/1060039726 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f906c004ff0 con 0x7f90600072d0 2024-03-29T11:34:38.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.749+0000 7f9071ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:38.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.749+0000 7f9071ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:34:38.747 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2024-03-29T11:34:38.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 10 librados: watch_flush enter 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 10 librados: watch_flush exit 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 10 monclient: shutdown 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 >> v1:172.21.15.33:6789/0 conn(0x7f90600072d0 legacy=0x7f907c139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 shutdown_connections 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 >> 172.21.15.33:0/1060039726 conn(0x7f907c066b00 msgr2=0x7f907c13bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 shutdown_connections 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 1 -- 172.21.15.33:0/1060039726 wait complete. 2024-03-29T11:34:38.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:38.750+0000 7f9083738640 1 librados: shutdown 2024-03-29T11:34:38.760 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:34:38.760 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:34:38.760 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:38.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:34:38.788 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:38.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:34:38.855 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:38.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:34:38.921 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-29T11:34:38.921 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:34:38.988 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:38.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:34:39.016 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:39.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:34:39.083 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:39.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:34:39.149 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-29T11:34:39.150 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:34:39.162 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:39.165+0000 7fed89632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:39.183 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:39.186+0000 7f4f95610040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:39.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 3dcbfff5-b024-40da-ae53-b95e8ef368d6 0 2024-03-29T11:34:39.219 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:39.221+0000 7fed89632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:39.238 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:39.242+0000 7f4f95610040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.321+0000 7f359598d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.321+0000 7f359598d640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.321+0000 7f359598d640 10 monclient: monmap: 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:39.323315+0000 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:39.323315+0000 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.320 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:39.321 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:39.321 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 1 Processor -- start 2024-03-29T11:34:39.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 1 -- start start 2024-03-29T11:34:39.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient: init 2024-03-29T11:34:39.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient(hunting): picked mon.noname-b con 0x7f359009ac40 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient(hunting): picked mon.noname-a con 0x7f359009c870 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient(hunting): picked mon.noname-c con 0x7f35900b4360 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359009dec0 con 0x7f359009c870 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359009da30 con 0x7f35900b4360 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900b39c0 con 0x7f359009ac40 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.322+0000 7f359598d640 10 monclient(hunting): authenticate will time out at 868.624825s 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f35900b4360 0x7f359014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56190/0 (socket says 172.21.15.153:56190) 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358f7fe640 1 -- 172.21.15.153:0/1548383945 learned_addr learned my addr 172.21.15.153:0/1548383945 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092515557 0 0) 0x7f359009da30 con 0x7f35900b4360 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): my global_id is 4099 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3578003520 con 0x7f35900b4360 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332739975 0 0) 0x7f35900b39c0 con 0x7f359009ac40 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f359009da30 con 0x7f359009ac40 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648436903 0 0) 0x7f3580001110 con 0x7f359009c870 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35900b39c0 con 0x7f359009c870 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329560487 0 0) 0x7f3578003520 con 0x7f35900b4360 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 >> v1:172.21.15.33:6790/0 conn(0x7f359009ac40 legacy=0x7f35900b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 >> v1:172.21.15.33:6789/0 conn(0x7f359009c870 legacy=0x7f359013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 monclient: found mon.noname-c 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900b5160 con 0x7f35900b4360 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f359598d640 5 monclient: authenticate success, global_id 4099 2024-03-29T11:34:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3588001a10 con 0x7f35900b4360 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f359598d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: dump: 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f359598d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3588001bf0 con 0x7f35900b4360 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f359598d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 1 -- 172.21.15.153:0/1548383945 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3588002000 con 0x7f35900b4360 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f359598d640 4 set_mon_vals no callback set 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.323+0000 7f358dffb640 10 monclient: dump: 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 10 monclient: shutdown 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 1 -- 172.21.15.153:0/1548383945 >> v1:172.21.15.153:6789/0 conn(0x7f35900b4360 legacy=0x7f359014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 1 -- 172.21.15.153:0/1548383945 shutdown_connections 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 1 -- 172.21.15.153:0/1548383945 >> 172.21.15.153:0/1548383945 conn(0x7f3590066b00 msgr2=0x7f3590066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 1 -- 172.21.15.153:0/1548383945 shutdown_connections 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 1 -- 172.21.15.153:0/1548383945 wait complete. 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.324+0000 7f359598d640 10 monclient: monmap: 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.335 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 librados: starting msgr at 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 librados: starting objecter 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 Processor -- start 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 -- start start 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 librados: setting wanted keys 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 librados: calling monclient init 2024-03-29T11:34:39.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient: init 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient(hunting): picked mon.noname-c con 0x7f359009ac40 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient(hunting): picked mon.noname-a con 0x7f35901d8690 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient(hunting): picked mon.noname-b con 0x7f35901455c0 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35901d7af0 con 0x7f35901d8690 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3590145a10 con 0x7f35901455c0 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900b5a20 con 0x7f359009ac40 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.325+0000 7f359598d640 10 monclient(hunting): authenticate will time out at 868.627810s 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f358f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f35901455c0 0x7f35901e2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56192/0 (socket says 172.21.15.153:56192) 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f358f7fe640 1 -- 172.21.15.153:0/2431232732 learned_addr learned my addr 172.21.15.153:0/2431232732 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 client.?.objecter ms_handle_connect 0x7f35901455c0 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 client.?.objecter ms_handle_connect 0x7f35901d8690 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 client.?.objecter ms_handle_connect 0x7f359009ac40 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592437009 0 0) 0x7f3590145a10 con 0x7f35901455c0 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f357c004eb0 con 0x7f35901455c0 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395384329 0 0) 0x7f35901d7af0 con 0x7f35901d8690 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3590145a10 con 0x7f35901d8690 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464949819 0 0) 0x7f35900b5a20 con 0x7f359009ac40 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35901d7af0 con 0x7f359009ac40 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 90614651 0 0) 0x7f3590145a10 con 0x7f35901d8690 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35900b5a20 con 0x7f35901d8690 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3580001760 con 0x7f35901d8690 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): dump: 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.339 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 230427877 0 0) 0x7f357c004eb0 con 0x7f35901455c0 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3590145a10 con 0x7f35901455c0 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3588001f00 con 0x7f35901455c0 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): dump: 2024-03-29T11:34:39.340 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086397575 0 0) 0x7f35901d7af0 con 0x7f359009ac40 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f357c004eb0 con 0x7f359009ac40 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3584001870 con 0x7f359009ac40 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:39.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient(hunting): dump: 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1064667413 0 0) 0x7f35900b5a20 con 0x7f35901d8690 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 >> v1:172.21.15.33:6790/0 conn(0x7f359009ac40 legacy=0x7f35901d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 >> v1:172.21.15.153:6789/0 conn(0x7f35901455c0 legacy=0x7f35901e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 monclient: found mon.a 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3590141960 con 0x7f35901d8690 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 5 monclient: authenticate success, global_id 4119 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 10 monclient: _renew_subs 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35901e3b20 con 0x7f35901d8690 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 10 monclient: _renew_subs 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35901e3e90 con 0x7f35901d8690 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f359598d640 1 librados: init done 2024-03-29T11:34:39.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3580001bf0 con 0x7f35901d8690 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f35800037b0 con 0x7f35901d8690 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f358d7fa640 4 set_mon_vals no callback set 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.326+0000 7f3577fff640 10 monclient: dump: 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.344 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3580003ca0 con 0x7f35901d8690 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 606123503 0 0) 0x7f35800051e0 con 0x7f35901d8690 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f3577fff640 10 client.4119.objecter ms_dispatch 0x7f3590000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f3577fff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f3577fff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f3577fff640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f359598d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f359598d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f359598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.327+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570005080 con 0x7f35901d8690 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.329+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f35800040e0 con 0x7f35901d8690 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.329+0000 7f3577fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.329+0000 7f3577fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:39.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.402+0000 7f359598d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3dcbfff5-b024-40da-ae53-b95e8ef368d6", "id": 0}] 2024-03-29T11:34:39.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.402+0000 7f359598d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3dcbfff5-b024-40da-ae53-b95e8ef368d6", "id": 0}] 2024-03-29T11:34:39.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.402+0000 7f359598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:39.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.402+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3dcbfff5-b024-40da-ae53-b95e8ef368d6", "id": 0} v 0) v1 -- 0x7f35700058b0 con 0x7f35901d8690 2024-03-29T11:34:39.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.403+0000 7f3577fff640 1 -- 172.21.15.153:0/2431232732 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3dcbfff5-b024-40da-ae53-b95e8ef368d6", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3087716900 0 732707515) 0x7f3580004280 con 0x7f35901d8690 2024-03-29T11:34:39.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.403+0000 7f3577fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3dcbfff5-b024-40da-ae53-b95e8ef368d6", "id": 0}] 2024-03-29T11:34:39.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.403+0000 7f3577fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:39.402 INFO:teuthology.orchestra.run.smithi153.stdout:0 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 10 librados: watch_flush enter 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 10 librados: watch_flush exit 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 10 client.4119.objecter successfully canceled tick 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 10 monclient: shutdown 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 >> v1:172.21.15.33:6789/0 conn(0x7f35901d8690 legacy=0x7f3590144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 shutdown_connections 2024-03-29T11:34:39.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 >> 172.21.15.153:0/2431232732 conn(0x7f3590066b00 msgr2=0x7f359013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:39.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 shutdown_connections 2024-03-29T11:34:39.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.404+0000 7f359598d640 1 -- 172.21.15.153:0/2431232732 wait complete. 2024-03-29T11:34:39.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.405+0000 7f359598d640 1 librados: shutdown 2024-03-29T11:34:39.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 8966becd-6685-4b87-90fa-332cb668e814 1 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.514+0000 7fb43cf9d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.514+0000 7fb43cf9d640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.514+0000 7fb43cf9d640 10 monclient: monmap: 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:39.515650+0000 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:39.515650+0000 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.514+0000 7fb43cf9d640 1 Processor -- start 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.514+0000 7fb43cf9d640 1 -- start start 2024-03-29T11:34:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.514+0000 7fb43cf9d640 10 monclient: init 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb43809ac40 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb43809c870 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4380b4360 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43809dec0 con 0x7fb4380b4360 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43809da30 con 0x7fb43809ac40 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380b39c0 con 0x7fb43809c870 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43cf9d640 10 monclient(hunting): authenticate will time out at 868.817407s 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43659c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb43809ac40 0x7fb4380b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56194/0 (socket says 172.21.15.153:56194) 2024-03-29T11:34:39.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43659c640 1 -- 172.21.15.153:0/4067629093 learned_addr learned my addr 172.21.15.153:0/4067629093 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044364290 0 0) 0x7fb43809da30 con 0x7fb43809ac40 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43559a640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb424003520 con 0x7fb43809ac40 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.515+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591400116 0 0) 0x7fb4380b39c0 con 0x7fb43809c870 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb43809da30 con 0x7fb43809c870 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969169230 0 0) 0x7fb430001490 con 0x7fb4380b4360 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4380b39c0 con 0x7fb4380b4360 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262774653 0 0) 0x7fb424003520 con 0x7fb43809ac40 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 >> v1:172.21.15.33:6790/0 conn(0x7fb43809c870 legacy=0x7fb43813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 >> v1:172.21.15.33:6789/0 conn(0x7fb4380b4360 legacy=0x7fb43814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 monclient: found mon.noname-c 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4380b5160 con 0x7fb43809ac40 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb420001690 con 0x7fb43809ac40 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: dump: 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb420001870 con 0x7fb43809ac40 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 1 -- 172.21.15.153:0/4067629093 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb420001c80 con 0x7fb43809ac40 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 4 set_mon_vals no callback set 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43559a640 10 monclient: dump: 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.527 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 10 monclient: shutdown 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 1 -- 172.21.15.153:0/4067629093 >> v1:172.21.15.153:6789/0 conn(0x7fb43809ac40 legacy=0x7fb4380b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 1 -- 172.21.15.153:0/4067629093 shutdown_connections 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 1 -- 172.21.15.153:0/4067629093 >> 172.21.15.153:0/4067629093 conn(0x7fb438066b00 msgr2=0x7fb438066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 1 -- 172.21.15.153:0/4067629093 shutdown_connections 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.516+0000 7fb43cf9d640 1 -- 172.21.15.153:0/4067629093 wait complete. 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.517+0000 7fb43cf9d640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.517+0000 7fb43cf9d640 10 monclient: monmap: 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.528 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.517+0000 7fb43cf9d640 1 librados: starting msgr at 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.517+0000 7fb43cf9d640 1 librados: starting objecter 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 1 Processor -- start 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 1 -- start start 2024-03-29T11:34:39.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 1 librados: setting wanted keys 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 1 librados: calling monclient init 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient: init 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb43809ac40 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4380af1d0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb438139370 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4381d4be0 con 0x7fb43809ac40 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380b06a0 con 0x7fb438139370 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380b1910 con 0x7fb4380af1d0 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb43cf9d640 10 monclient(hunting): authenticate will time out at 868.820471s 2024-03-29T11:34:39.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb436d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb438139370 0x7fb4381d70e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56204/0 (socket says 172.21.15.153:56204) 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb436d9d640 1 -- 172.21.15.153:0/111832247 learned_addr learned my addr 172.21.15.153:0/111832247 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 client.?.objecter ms_handle_connect 0x7fb438139370 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 client.?.objecter ms_handle_connect 0x7fb43809ac40 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 client.?.objecter ms_handle_connect 0x7fb4380af1d0 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213393069 0 0) 0x7fb4380b06a0 con 0x7fb438139370 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb418004eb0 con 0x7fb438139370 2024-03-29T11:34:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.518+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611194364 0 0) 0x7fb4381d4be0 con 0x7fb43809ac40 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4380b06a0 con 0x7fb43809ac40 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001359027 0 0) 0x7fb4380b1910 con 0x7fb4380af1d0 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4381d4be0 con 0x7fb4380af1d0 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326698426 0 0) 0x7fb418004eb0 con 0x7fb438139370 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4380b1910 con 0x7fb438139370 2024-03-29T11:34:39.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb430001bc0 con 0x7fb438139370 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): dump: 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1010829021 0 0) 0x7fb4380b06a0 con 0x7fb43809ac40 2024-03-29T11:34:39.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb418004eb0 con 0x7fb43809ac40 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb420001b80 con 0x7fb43809ac40 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): dump: 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022084874 0 0) 0x7fb4381d4be0 con 0x7fb4380af1d0 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4380b06a0 con 0x7fb4380af1d0 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb428001870 con 0x7fb4380af1d0 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient(hunting): dump: 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.535 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2843626707 0 0) 0x7fb4380b1910 con 0x7fb438139370 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 >> v1:172.21.15.33:6790/0 conn(0x7fb4380af1d0 legacy=0x7fb438138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 >> v1:172.21.15.33:6789/0 conn(0x7fb43809ac40 legacy=0x7fb4380afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 monclient: found mon.b 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438139970 con 0x7fb438139370 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 10 monclient: _renew_subs 2024-03-29T11:34:39.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4381d77f0 con 0x7fb438139370 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 10 monclient: _renew_subs 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4381d7d70 con 0x7fb438139370 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb43cf9d640 1 librados: init done 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb430002050 con 0x7fb438139370 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb430003b30 con 0x7fb438139370 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 10 monclient: dump: 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.537 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb434d99640 4 set_mon_vals no callback set 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.519+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb4300040a0 con 0x7fb438139370 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb43cf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb43cf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 108950317 0 0) 0x7fb430006720 con 0x7fb438139370 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb43cf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb42effd640 10 client.4108.objecter ms_dispatch 0x7fb438000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:34:39.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb42effd640 3 client.4108.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:34:39.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb42effd640 3 client.4108.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:34:39.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c005080 con 0x7fb438139370 2024-03-29T11:34:39.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.520+0000 7fb42effd640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-29T11:34:39.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.522+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb430004580 con 0x7fb438139370 2024-03-29T11:34:39.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.522+0000 7fb42effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.522+0000 7fb42effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.595+0000 7fb43cf9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8966becd-6685-4b87-90fa-332cb668e814", "id": 1}] 2024-03-29T11:34:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.595+0000 7fb43cf9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8966becd-6685-4b87-90fa-332cb668e814", "id": 1}] 2024-03-29T11:34:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.595+0000 7fb43cf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.595+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8966becd-6685-4b87-90fa-332cb668e814", "id": 1} v 0) v1 -- 0x7fb41c0058b0 con 0x7fb438139370 2024-03-29T11:34:39.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.603+0000 7fb42effd640 1 -- 172.21.15.153:0/111832247 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8966becd-6685-4b87-90fa-332cb668e814", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2318773091 0 940483276) 0x7fb430004720 con 0x7fb438139370 2024-03-29T11:34:39.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.603+0000 7fb42effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8966becd-6685-4b87-90fa-332cb668e814", "id": 1}] 2024-03-29T11:34:39.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.603+0000 7fb42effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:39.601 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-03-29T11:34:39.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 10 librados: watch_flush enter 2024-03-29T11:34:39.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 10 librados: watch_flush exit 2024-03-29T11:34:39.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-29T11:34:39.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 10 client.4108.objecter successfully canceled tick 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 10 monclient: shutdown 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 >> v1:172.21.15.153:6789/0 conn(0x7fb438139370 legacy=0x7fb4381d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 shutdown_connections 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 >> 172.21.15.153:0/111832247 conn(0x7fb438066b00 msgr2=0x7fb4380b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 shutdown_connections 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 1 -- 172.21.15.153:0/111832247 wait complete. 2024-03-29T11:34:39.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.604+0000 7fb43cf9d640 1 librados: shutdown 2024-03-29T11:34:39.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 5b574eda-e056-425d-9498-efa05d611207 2 2024-03-29T11:34:39.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.719+0000 7f96df8d3640 10 monclient: get_monmap_and_config 2024-03-29T11:34:39.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.719+0000 7f96df8d3640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.719+0000 7f96df8d3640 10 monclient: monmap: 2024-03-29T11:34:39.717 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.717 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.717 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:39.720890+0000 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:39.720890+0000 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.719+0000 7f96df8d3640 1 Processor -- start 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 1 -- start start 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient: init 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient(hunting): picked mon.noname-b con 0x7f96d809ac40 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient(hunting): picked mon.noname-a con 0x7f96d809c870 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient(hunting): picked mon.noname-c con 0x7f96d80b4360 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d809dec0 con 0x7f96d809c870 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d809da30 con 0x7f96d80b4360 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d80b39c0 con 0x7f96d809ac40 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96df8d3640 10 monclient(hunting): authenticate will time out at 869.022567s 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96dde70640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f96d80b4360 0x7f96d814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56206/0 (socket says 172.21.15.153:56206) 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.720+0000 7f96dde70640 1 -- 172.21.15.153:0/2736568399 learned_addr learned my addr 172.21.15.153:0/2736568399 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064330018 0 0) 0x7f96d809da30 con 0x7f96d80b4360 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c4003520 con 0x7f96d80b4360 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053898806 0 0) 0x7f96d80b39c0 con 0x7f96d809ac40 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d809da30 con 0x7f96d809ac40 2024-03-29T11:34:39.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682461101 0 0) 0x7f96c8001110 con 0x7f96d809c870 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d80b39c0 con 0x7f96d809c870 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1026463302 0 0) 0x7f96c4003520 con 0x7f96d80b4360 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 >> v1:172.21.15.33:6790/0 conn(0x7f96d809ac40 legacy=0x7f96d80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 >> v1:172.21.15.33:6789/0 conn(0x7f96d809c870 legacy=0x7f96d813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 monclient: found mon.noname-c 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d80b5160 con 0x7f96d80b4360 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f96d0001a10 con 0x7f96d80b4360 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96df8d3640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96df8d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: dump: 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96df8d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96d0001bf0 con 0x7f96d80b4360 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 1 -- 172.21.15.153:0/2736568399 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f96d0002000 con 0x7f96d80b4360 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96cffff640 10 monclient: dump: 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.731 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96df8d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.721+0000 7f96df8d3640 4 set_mon_vals no callback set 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 10 monclient: shutdown 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 1 -- 172.21.15.153:0/2736568399 >> v1:172.21.15.153:6789/0 conn(0x7f96d80b4360 legacy=0x7f96d814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 1 -- 172.21.15.153:0/2736568399 shutdown_connections 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 1 -- 172.21.15.153:0/2736568399 >> 172.21.15.153:0/2736568399 conn(0x7f96d8066b00 msgr2=0x7f96d8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 1 -- 172.21.15.153:0/2736568399 shutdown_connections 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 1 -- 172.21.15.153:0/2736568399 wait complete. 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.722+0000 7f96df8d3640 10 monclient: monmap: 2024-03-29T11:34:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 librados: starting msgr at 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 librados: starting objecter 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 Processor -- start 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 -- start start 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 librados: setting wanted keys 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 librados: calling monclient init 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient: init 2024-03-29T11:34:39.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient(hunting): picked mon.noname-c con 0x7f96d809ac40 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient(hunting): picked mon.noname-a con 0x7f96d81d8690 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient(hunting): picked mon.noname-b con 0x7f96d81455c0 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d81d7af0 con 0x7f96d81d8690 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d8145a10 con 0x7f96d81455c0 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d80b5a20 con 0x7f96d809ac40 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96df8d3640 10 monclient(hunting): authenticate will time out at 869.025567s 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96dde70640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f96d81455c0 0x7f96d81e2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56212/0 (socket says 172.21.15.153:56212) 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96dde70640 1 -- 172.21.15.153:0/1076021202 learned_addr learned my addr 172.21.15.153:0/1076021202 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96cdffb640 10 client.?.objecter ms_handle_connect 0x7f96d81455c0 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96cdffb640 10 client.?.objecter ms_handle_connect 0x7f96d809ac40 2024-03-29T11:34:39.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96cdffb640 10 client.?.objecter ms_handle_connect 0x7f96d81d8690 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.723+0000 7f96cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768168149 0 0) 0x7f96d8145a10 con 0x7f96d81455c0 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96bc004eb0 con 0x7f96d81455c0 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854638771 0 0) 0x7f96d80b5a20 con 0x7f96d809ac40 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d8145a10 con 0x7f96d809ac40 2024-03-29T11:34:39.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180038829 0 0) 0x7f96d81d7af0 con 0x7f96d81d8690 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d80b5a20 con 0x7f96d81d8690 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1697245440 0 0) 0x7f96bc004eb0 con 0x7f96d81455c0 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96d81d7af0 con 0x7f96d81455c0 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f96d0001f40 con 0x7f96d81455c0 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): dump: 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.736 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 577770484 0 0) 0x7f96d8145a10 con 0x7f96d809ac40 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96bc004eb0 con 0x7f96d809ac40 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f96d4001970 con 0x7f96d809ac40 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): dump: 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.737 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4162802942 0 0) 0x7f96d80b5a20 con 0x7f96d81d8690 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96d8145a10 con 0x7f96d81d8690 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f96c8001760 con 0x7f96d81d8690 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient(hunting): dump: 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.738 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 519969167 0 0) 0x7f96d81d7af0 con 0x7f96d81455c0 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 >> v1:172.21.15.33:6790/0 conn(0x7f96d809ac40 legacy=0x7f96d81d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 >> v1:172.21.15.33:6789/0 conn(0x7f96d81d8690 legacy=0x7f96d8144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 monclient: found mon.b 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d8141960 con 0x7f96d81455c0 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 10 monclient: _renew_subs 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d81e3b20 con 0x7f96d81455c0 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96d0004e10 con 0x7f96d81455c0 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f96d0005120 con 0x7f96d81455c0 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 monclient: dump: 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.740 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f96d0005610 con 0x7f96d81455c0 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cf7fe640 4 set_mon_vals no callback set 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 10 monclient: _renew_subs 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d81e3e90 con 0x7f96d81455c0 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96df8d3640 1 librados: init done 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 208165276 0 0) 0x7f96d0005f10 con 0x7f96d81455c0 2024-03-29T11:34:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 10 client.4114.objecter ms_dispatch 0x7f96d8000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 3 client.4114.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.724+0000 7f96cdffb640 3 client.4114.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.725+0000 7f96cdffb640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.725+0000 7f96df8d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.725+0000 7f96df8d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.725+0000 7f96df8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.725+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d8141b50 con 0x7f96d81455c0 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.727+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f96d00060d0 con 0x7f96d81455c0 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.727+0000 7f96cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.727+0000 7f96cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:39.771 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:39.773+0000 7fed89632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:39.782 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:39.785+0000 7f4f95610040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:39.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.798+0000 7f96df8d3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5b574eda-e056-425d-9498-efa05d611207", "id": 2}] 2024-03-29T11:34:39.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.798+0000 7f96df8d3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5b574eda-e056-425d-9498-efa05d611207", "id": 2}] 2024-03-29T11:34:39.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.798+0000 7f96df8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.798+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5b574eda-e056-425d-9498-efa05d611207", "id": 2} v 0) v1 -- 0x7f96d804f730 con 0x7f96d81455c0 2024-03-29T11:34:39.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.806+0000 7f96cdffb640 1 -- 172.21.15.153:0/1076021202 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5b574eda-e056-425d-9498-efa05d611207", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1359038158 0 216599125) 0x7f96d0005af0 con 0x7f96d81455c0 2024-03-29T11:34:39.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.806+0000 7f96cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5b574eda-e056-425d-9498-efa05d611207", "id": 2}] 2024-03-29T11:34:39.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.806+0000 7f96cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:39.804 INFO:teuthology.orchestra.run.smithi153.stdout:2 2024-03-29T11:34:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.806+0000 7f96df8d3640 10 librados: watch_flush enter 2024-03-29T11:34:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 10 librados: watch_flush exit 2024-03-29T11:34:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:34:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 10 monclient: shutdown 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 >> v1:172.21.15.153:6789/0 conn(0x7f96d81455c0 legacy=0x7f96d81e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 shutdown_connections 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 >> 172.21.15.153:0/1076021202 conn(0x7f96d8066b00 msgr2=0x7f96d813a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 shutdown_connections 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 1 -- 172.21.15.153:0/1076021202 wait complete. 2024-03-29T11:34:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.807+0000 7f96df8d3640 1 librados: shutdown 2024-03-29T11:34:39.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new d95fd6fd-9fcb-419f-9953-c0d0fa896631 3 2024-03-29T11:34:39.827 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:39.829+0000 7fed89632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:39.838 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:39.842+0000 7f4f95610040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:39.874 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:39.876+0000 7fed89632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:39.885 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:39.888+0000 7f4f95610040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.917+0000 7f5042af8640 10 monclient: get_monmap_and_config 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.917+0000 7f5042af8640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.917+0000 7f5042af8640 10 monclient: monmap: 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:39.918868+0000 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:39.918868+0000 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.915 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.916 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:39.916 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:39.916 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 1 Processor -- start 2024-03-29T11:34:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 1 -- start start 2024-03-29T11:34:39.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient: init 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient(hunting): picked mon.noname-a con 0x7f503c09ac40 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient(hunting): picked mon.noname-b con 0x7f503c09c870 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient(hunting): picked mon.noname-c con 0x7f503c0b4360 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c09dec0 con 0x7f503c09ac40 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c09da30 con 0x7f503c0b4360 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b39c0 con 0x7f503c09c870 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5042af8640 10 monclient(hunting): authenticate will time out at 869.220621s 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5041095640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f503c0b4360 0x7f503c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56218/0 (socket says 172.21.15.153:56218) 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.918+0000 7f5041095640 1 -- 172.21.15.153:0/2262164567 learned_addr learned my addr 172.21.15.153:0/2262164567 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262350711 0 0) 0x7f503c09da30 con 0x7f503c0b4360 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c003520 con 0x7f503c0b4360 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736451306 0 0) 0x7f503c0b39c0 con 0x7f503c09c870 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c09da30 con 0x7f503c09c870 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465860087 0 0) 0x7f5030001040 con 0x7f503c09ac40 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:34:39.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c0b39c0 con 0x7f503c09ac40 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806290971 0 0) 0x7f502c003520 con 0x7f503c0b4360 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 >> v1:172.21.15.33:6790/0 conn(0x7f503c09c870 legacy=0x7f503c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 >> v1:172.21.15.33:6789/0 conn(0x7f503c09ac40 legacy=0x7f503c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0b5160 con 0x7f503c0b4360 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f5042af8640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5034001a10 con 0x7f503c0b4360 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:39.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: dump: 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f5042af8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5034001bf0 con 0x7f503c0b4360 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 1 -- 172.21.15.153:0/2262164567 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5034002000 con 0x7f503c0b4360 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f503b7fe640 10 monclient: dump: 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f5042af8640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.919+0000 7f5042af8640 4 set_mon_vals no callback set 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 10 monclient: shutdown 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 1 -- 172.21.15.153:0/2262164567 >> v1:172.21.15.153:6789/0 conn(0x7f503c0b4360 legacy=0x7f503c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 1 -- 172.21.15.153:0/2262164567 shutdown_connections 2024-03-29T11:34:39.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 1 -- 172.21.15.153:0/2262164567 >> 172.21.15.153:0/2262164567 conn(0x7f503c066b00 msgr2=0x7f503c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 1 -- 172.21.15.153:0/2262164567 shutdown_connections 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 1 -- 172.21.15.153:0/2262164567 wait complete. 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.920+0000 7f5042af8640 10 monclient: build_initial_monmap 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 10 monclient: monmap: 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:39.930 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 1 librados: starting msgr at 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 1 librados: starting objecter 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 1 Processor -- start 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 1 -- start start 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 1 librados: setting wanted keys 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 1 librados: calling monclient init 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.921+0000 7f5042af8640 10 monclient: init 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient(hunting): picked mon.noname-b con 0x7f503c09ac40 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient(hunting): picked mon.noname-c con 0x7f503c0aee50 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:39.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient(hunting): picked mon.noname-a con 0x7f503c0ae810 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c1d7b20 con 0x7f503c0ae810 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b00d0 con 0x7f503c09ac40 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0ada70 con 0x7f503c0aee50 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient(hunting): authenticate will time out at 869.223965s 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5040894640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f503c09ac40 0x7f503c1d7fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56228/0 (socket says 172.21.15.153:56228) 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5040894640 1 -- 172.21.15.153:0/435757352 learned_addr learned my addr 172.21.15.153:0/435757352 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 client.?.objecter ms_handle_connect 0x7f503c09ac40 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 client.?.objecter ms_handle_connect 0x7f503c0ae810 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 client.?.objecter ms_handle_connect 0x7f503c0aee50 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567596170 0 0) 0x7f503c0b00d0 con 0x7f503c09ac40 2024-03-29T11:34:39.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5020004eb0 con 0x7f503c09ac40 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168421400 0 0) 0x7f503c0ada70 con 0x7f503c0aee50 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c0b00d0 con 0x7f503c0aee50 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807935806 0 0) 0x7f503c1d7b20 con 0x7f503c0ae810 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:34:39.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c0ada70 con 0x7f503c0ae810 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447327255 0 0) 0x7f5020004eb0 con 0x7f503c09ac40 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f503c1d7b20 con 0x7f503c09ac40 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5030001810 con 0x7f503c09ac40 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): dump: 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.934 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1290177995 0 0) 0x7f503c0b00d0 con 0x7f503c0aee50 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5020004eb0 con 0x7f503c0aee50 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5028001930 con 0x7f503c0aee50 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): dump: 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.935 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1751586181 0 0) 0x7f503c0ada70 con 0x7f503c0ae810 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f503c0b00d0 con 0x7f503c0ae810 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5034001d60 con 0x7f503c0ae810 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient(hunting): dump: 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.936 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3125564343 0 0) 0x7f503c1d7b20 con 0x7f503c09ac40 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 >> v1:172.21.15.33:6790/0 conn(0x7f503c0aee50 legacy=0x7f503c0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 >> v1:172.21.15.33:6789/0 conn(0x7f503c0ae810 legacy=0x7f503c0b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 monclient: found mon.b 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c1d7640 con 0x7f503c09ac40 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:39.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f50397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient: _renew_subs 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.922+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c0b17b0 con 0x7f503c09ac40 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 10 monclient: _renew_subs 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c0b1b50 con 0x7f503c09ac40 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 1 librados: init done 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5030003610 con 0x7f503c09ac40 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5030001d80 con 0x7f503c09ac40 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:39.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 10 monclient: dump: 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f503affd640 4 set_mon_vals no callback set 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5030003db0 con 0x7f503c09ac40 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3434008310 0 0) 0x7f50300065f0 con 0x7f503c09ac40 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 10 client.4120.objecter ms_dispatch 0x7f503c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:34:39.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 3 client.4120.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 3 client.4120.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f50397fa640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.923+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501c005080 con 0x7f503c09ac40 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.925+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5030004290 con 0x7f503c09ac40 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.925+0000 7f50397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:39.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.925+0000 7f50397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:39.941 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:39.934+0000 7fed89632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:39.941 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:39.943+0000 7f4f95610040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:39.975 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:39.977+0000 7fed89632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:39.984 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:39.987+0000 7f4f95610040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:39.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.997+0000 7f5042af8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d95fd6fd-9fcb-419f-9953-c0d0fa896631", "id": 3}] 2024-03-29T11:34:39.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.997+0000 7f5042af8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d95fd6fd-9fcb-419f-9953-c0d0fa896631", "id": 3}] 2024-03-29T11:34:39.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.997+0000 7f5042af8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:39.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:39.997+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d95fd6fd-9fcb-419f-9953-c0d0fa896631", "id": 3} v 0) v1 -- 0x7f501c0058b0 con 0x7f503c09ac40 2024-03-29T11:34:40.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.003+0000 7f50397fa640 1 -- 172.21.15.153:0/435757352 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d95fd6fd-9fcb-419f-9953-c0d0fa896631", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 644492587 0 525046306) 0x7f5030006860 con 0x7f503c09ac40 2024-03-29T11:34:40.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.003+0000 7f50397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d95fd6fd-9fcb-419f-9953-c0d0fa896631", "id": 3}] 2024-03-29T11:34:40.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.003+0000 7f50397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:40.001 INFO:teuthology.orchestra.run.smithi153.stdout:3 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 10 librados: watch_flush enter 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 10 librados: watch_flush exit 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 10 monclient: shutdown 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 >> v1:172.21.15.153:6789/0 conn(0x7f503c09ac40 legacy=0x7f503c1d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 shutdown_connections 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 >> 172.21.15.153:0/435757352 conn(0x7f503c066b00 msgr2=0x7f503c13a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 shutdown_connections 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 1 -- 172.21.15.153:0/435757352 wait complete. 2024-03-29T11:34:40.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.004+0000 7f5042af8640 1 librados: shutdown 2024-03-29T11:34:40.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 355b699b-9a4c-4f8d-a4d0-41e3ad19559a 4 2024-03-29T11:34:40.096 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:40.098+0000 7fed89632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:40.104 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:40.107+0000 7f4f95610040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.155+0000 7f53799c6640 10 monclient: get_monmap_and_config 2024-03-29T11:34:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.155+0000 7f53799c6640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.155+0000 7f53799c6640 10 monclient: monmap: 2024-03-29T11:34:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:40.157269+0000 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:40.157269+0000 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:40.154 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.156+0000 7f53799c6640 1 Processor -- start 2024-03-29T11:34:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.156+0000 7f53799c6640 1 -- start start 2024-03-29T11:34:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.156+0000 7f53799c6640 10 monclient: init 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient(hunting): picked mon.noname-a con 0x7f53740b8240 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient(hunting): picked mon.noname-c con 0x7f53740b1f60 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient(hunting): picked mon.noname-b con 0x7f537409c8c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537409bd80 con 0x7f53740b8240 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537409b960 con 0x7f53740b1f60 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53740b2ec0 con 0x7f537409c8c0 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient(hunting): authenticate will time out at 869.458977s 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f536bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f53740b1f60 0x7f537409c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56242/0 (socket says 172.21.15.153:56242) 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f536bfff640 1 -- 172.21.15.153:0/2446678019 learned_addr learned my addr 172.21.15.153:0/2446678019 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272792628 0 0) 0x7f537409b960 con 0x7f53740b1f60 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5360003520 con 0x7f53740b1f60 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724400835 0 0) 0x7f537409bd80 con 0x7f53740b8240 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537409b960 con 0x7f53740b8240 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725341106 0 0) 0x7f53740b2ec0 con 0x7f537409c8c0 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537409bd80 con 0x7f537409c8c0 2024-03-29T11:34:40.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2185354672 0 0) 0x7f5360003520 con 0x7f53740b1f60 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 >> v1:172.21.15.33:6790/0 conn(0x7f537409c8c0 legacy=0x7f537414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 >> v1:172.21.15.33:6789/0 conn(0x7f53740b8240 legacy=0x7f537413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 monclient: found mon.noname-c 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537413fb60 con 0x7f53740b1f60 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f535c001690 con 0x7f53740b1f60 2024-03-29T11:34:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: dump: 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f535c001870 con 0x7f53740b1f60 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 1 -- 172.21.15.153:0/2446678019 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f535c001c80 con 0x7f53740b1f60 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53727fc640 10 monclient: dump: 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.167 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.157+0000 7f53799c6640 4 set_mon_vals no callback set 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.158+0000 7f53799c6640 10 monclient: shutdown 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.158+0000 7f53799c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.158+0000 7f53799c6640 1 -- 172.21.15.153:0/2446678019 >> v1:172.21.15.153:6789/0 conn(0x7f53740b1f60 legacy=0x7f537409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.158+0000 7f53799c6640 1 -- 172.21.15.153:0/2446678019 shutdown_connections 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.158+0000 7f53799c6640 1 -- 172.21.15.153:0/2446678019 >> 172.21.15.153:0/2446678019 conn(0x7f5374066b00 msgr2=0x7f5374066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.158+0000 7f53799c6640 1 -- 172.21.15.153:0/2446678019 shutdown_connections 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.158+0000 7f53799c6640 1 -- 172.21.15.153:0/2446678019 wait complete. 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 10 monclient: monmap: 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.168 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 1 librados: starting msgr at 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 1 librados: starting objecter 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 1 Processor -- start 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 1 -- start start 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 1 librados: setting wanted keys 2024-03-29T11:34:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 1 librados: calling monclient init 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.159+0000 7f53799c6640 10 monclient: init 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient(hunting): picked mon.noname-b con 0x7f537409c8c0 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient(hunting): picked mon.noname-c con 0x7f53740b8240 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient(hunting): picked mon.noname-a con 0x7f5374144bf0 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5374131f80 con 0x7f5374144bf0 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5374133010 con 0x7f537409c8c0 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53741332f0 con 0x7f53740b8240 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient(hunting): authenticate will time out at 869.461998s 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5372ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f537409c8c0 0x7f5374131720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56256/0 (socket says 172.21.15.153:56256) 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5372ffd640 1 -- 172.21.15.153:0/3408453116 learned_addr learned my addr 172.21.15.153:0/3408453116 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 client.?.objecter ms_handle_connect 0x7f537409c8c0 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 client.?.objecter ms_handle_connect 0x7f53740b8240 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 client.?.objecter ms_handle_connect 0x7f5374144bf0 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542579787 0 0) 0x7f5374133010 con 0x7f537409c8c0 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5354004eb0 con 0x7f537409c8c0 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707917158 0 0) 0x7f53741332f0 con 0x7f53740b8240 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5374133010 con 0x7f53740b8240 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504031049 0 0) 0x7f5374131f80 con 0x7f5374144bf0 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53741332f0 con 0x7f5374144bf0 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681532994 0 0) 0x7f5354004eb0 con 0x7f537409c8c0 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5374131f80 con 0x7f537409c8c0 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5364001a30 con 0x7f537409c8c0 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:40.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1972154888 0 0) 0x7f5374133010 con 0x7f53740b8240 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5354004eb0 con 0x7f53740b8240 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f535c001b80 con 0x7f53740b8240 2024-03-29T11:34:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1527855910 0 0) 0x7f53741332f0 con 0x7f5374144bf0 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5374133010 con 0x7f5374144bf0 2024-03-29T11:34:40.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f536c001a30 con 0x7f5374144bf0 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2686553441 0 0) 0x7f5374131f80 con 0x7f537409c8c0 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 >> v1:172.21.15.33:6790/0 conn(0x7f53740b8240 legacy=0x7f53741444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 >> v1:172.21.15.33:6789/0 conn(0x7f5374144bf0 legacy=0x7f5374145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 monclient: found mon.b 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5374132410 con 0x7f537409c8c0 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f5370ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient: _renew_subs 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.160+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5374131190 con 0x7f537409c8c0 2024-03-29T11:34:40.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 10 monclient: _renew_subs 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5374145a10 con 0x7f537409c8c0 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 1 librados: init done 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5364001c70 con 0x7f537409c8c0 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5364003a60 con 0x7f537409c8c0 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5371ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 10 monclient: dump: 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.177 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5364003fd0 con 0x7f537409c8c0 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1048809880 0 0) 0x7f53640067a0 con 0x7f537409c8c0 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 10 client.4126.objecter ms_dispatch 0x7f5374000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 3 client.4126.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:34:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 3 client.4126.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f5370ff9640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.161+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5358005080 con 0x7f537409c8c0 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.163+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f53640061d0 con 0x7f537409c8c0 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.163+0000 7f5370ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.163+0000 7f5370ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:40.179 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:40.172+0000 7fed89632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:40.180 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:40.180+0000 7f4f95610040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:40.216 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:40.218+0000 7fed89632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:40.224 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:40.227+0000 7f4f95610040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.234+0000 7f53799c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "355b699b-9a4c-4f8d-a4d0-41e3ad19559a", "id": 4}] 2024-03-29T11:34:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.234+0000 7f53799c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "355b699b-9a4c-4f8d-a4d0-41e3ad19559a", "id": 4}] 2024-03-29T11:34:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.234+0000 7f53799c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.234+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "355b699b-9a4c-4f8d-a4d0-41e3ad19559a", "id": 4} v 0) v1 -- 0x7f53580058b0 con 0x7f537409c8c0 2024-03-29T11:34:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.238+0000 7f5370ff9640 1 -- 172.21.15.153:0/3408453116 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "355b699b-9a4c-4f8d-a4d0-41e3ad19559a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3815644165 0 1697012583) 0x7f5364006370 con 0x7f537409c8c0 2024-03-29T11:34:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.238+0000 7f5370ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "355b699b-9a4c-4f8d-a4d0-41e3ad19559a", "id": 4}] 2024-03-29T11:34:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.238+0000 7f5370ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:40.236 INFO:teuthology.orchestra.run.smithi153.stdout:4 2024-03-29T11:34:40.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 10 librados: watch_flush enter 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 10 librados: watch_flush exit 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 10 monclient: shutdown 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 >> v1:172.21.15.153:6789/0 conn(0x7f537409c8c0 legacy=0x7f5374131720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 shutdown_connections 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 >> 172.21.15.153:0/3408453116 conn(0x7f5374066b00 msgr2=0x7f537413dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 shutdown_connections 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 1 -- 172.21.15.153:0/3408453116 wait complete. 2024-03-29T11:34:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.239+0000 7f53799c6640 1 librados: shutdown 2024-03-29T11:34:40.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 057e8b01-3785-482b-8261-8b53a0386027 5 2024-03-29T11:34:40.270 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:40.273+0000 7fed89632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:40.279 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:40.282+0000 7f4f95610040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:40.320 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:40.322+0000 7fed89632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:40.328 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:40.331+0000 7f4f95610040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:40.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.371+0000 7fbcc004e640 10 monclient: get_monmap_and_config 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.371+0000 7fbcc004e640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.371+0000 7fbcc004e640 10 monclient: monmap: 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:40.372692+0000 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:40.372692+0000 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:40.369 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.371+0000 7fbcc004e640 1 Processor -- start 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.371+0000 7fbcc004e640 1 -- start start 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.371+0000 7fbcc004e640 10 monclient: init 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 10 monclient(hunting): picked mon.noname-a con 0x7fbcb809ac40 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 10 monclient(hunting): picked mon.noname-b con 0x7fbcb809c870 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 10 monclient(hunting): picked mon.noname-c con 0x7fbcb80b4360 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb809dec0 con 0x7fbcb809ac40 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb809da30 con 0x7fbcb80b4360 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb80b39c0 con 0x7fbcb809c870 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcc004e640 10 monclient(hunting): authenticate will time out at 869.674457s 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcbe5eb640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbcb80b4360 0x7fbcb814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56268/0 (socket says 172.21.15.153:56268) 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.372+0000 7fbcbe5eb640 1 -- 172.21.15.153:0/2891144371 learned_addr learned my addr 172.21.15.153:0/2891144371 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627663542 0 0) 0x7fbcb809da30 con 0x7fbcb80b4360 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca4003520 con 0x7fbcb80b4360 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933858494 0 0) 0x7fbcb80b39c0 con 0x7fbcb809c870 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb809da30 con 0x7fbcb809c870 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653764831 0 0) 0x7fbca8001040 con 0x7fbcb809ac40 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb80b39c0 con 0x7fbcb809ac40 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3177844459 0 0) 0x7fbca4003520 con 0x7fbcb80b4360 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 >> v1:172.21.15.33:6790/0 conn(0x7fbcb809c870 legacy=0x7fbcb813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 >> v1:172.21.15.33:6789/0 conn(0x7fbcb809ac40 legacy=0x7fbcb80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 monclient: found mon.noname-c 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb80b5160 con 0x7fbcb80b4360 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbcb0001a10 con 0x7fbcb80b4360 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcc004e640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcc004e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: dump: 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.382 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcc004e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcb0001bf0 con 0x7fbcb80b4360 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcc004e640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 1 -- 172.21.15.153:0/2891144371 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbcb0002000 con 0x7fbcb80b4360 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcc004e640 4 set_mon_vals no callback set 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcbcde8640 10 monclient: dump: 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.383 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.373+0000 7fbcc004e640 10 monclient: shutdown 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 1 -- 172.21.15.153:0/2891144371 >> v1:172.21.15.153:6789/0 conn(0x7fbcb80b4360 legacy=0x7fbcb814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 1 -- 172.21.15.153:0/2891144371 shutdown_connections 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 1 -- 172.21.15.153:0/2891144371 >> 172.21.15.153:0/2891144371 conn(0x7fbcb8066b00 msgr2=0x7fbcb8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 1 -- 172.21.15.153:0/2891144371 shutdown_connections 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 1 -- 172.21.15.153:0/2891144371 wait complete. 2024-03-29T11:34:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 10 monclient: monmap: 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 1 librados: starting msgr at 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.374+0000 7fbcc004e640 1 librados: starting objecter 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 1 Processor -- start 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 1 -- start start 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 1 librados: setting wanted keys 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 1 librados: calling monclient init 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient: init 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient(hunting): picked mon.noname-c con 0x7fbcb809ac40 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient(hunting): picked mon.noname-b con 0x7fbcb80aee50 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient(hunting): picked mon.noname-a con 0x7fbcb80ae810 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb81d7b20 con 0x7fbcb80ae810 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb80b00d0 con 0x7fbcb80aee50 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb80ada70 con 0x7fbcb809ac40 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcc004e640 10 monclient(hunting): authenticate will time out at 869.677502s 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcbd5e9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbcb80aee50 0x7fbcb80b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56270/0 (socket says 172.21.15.153:56270) 2024-03-29T11:34:40.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcbd5e9640 1 -- 172.21.15.153:0/1444342577 learned_addr learned my addr 172.21.15.153:0/1444342577 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 client.?.objecter ms_handle_connect 0x7fbcb80aee50 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 client.?.objecter ms_handle_connect 0x7fbcb809ac40 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 client.?.objecter ms_handle_connect 0x7fbcb80ae810 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786268173 0 0) 0x7fbcb80b00d0 con 0x7fbcb80aee50 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.375+0000 7fbcae7fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c004eb0 con 0x7fbcb80aee50 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194433894 0 0) 0x7fbcb80ada70 con 0x7fbcb809ac40 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb80b00d0 con 0x7fbcb809ac40 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875233349 0 0) 0x7fbcb81d7b20 con 0x7fbcb80ae810 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb80ada70 con 0x7fbcb80ae810 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1537579442 0 0) 0x7fbc9c004eb0 con 0x7fbcb80aee50 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcb81d7b20 con 0x7fbcb80aee50 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbcb4001a30 con 0x7fbcb80aee50 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1326326001 0 0) 0x7fbcb80b00d0 con 0x7fbcb809ac40 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc9c004eb0 con 0x7fbcb809ac40 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbca80016d0 con 0x7fbcb809ac40 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:40.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.390 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2574036276 0 0) 0x7fbcb80ada70 con 0x7fbcb80ae810 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcb80b00d0 con 0x7fbcb80ae810 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbcb0001f00 con 0x7fbcb80ae810 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.391 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2532780357 0 0) 0x7fbcb81d7b20 con 0x7fbcb80aee50 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 >> v1:172.21.15.33:6790/0 conn(0x7fbcb809ac40 legacy=0x7fbcb81d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 >> v1:172.21.15.33:6789/0 conn(0x7fbcb80ae810 legacy=0x7fbcb80b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 monclient: found mon.b 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb81d7640 con 0x7fbcb80aee50 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 10 monclient: _renew_subs 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb80b17b0 con 0x7fbcb80aee50 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 10 monclient: _renew_subs 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb81e3e90 con 0x7fbcb80aee50 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcc004e640 1 librados: init done 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcb4001c70 con 0x7fbcb80aee50 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbcb4003a60 con 0x7fbcb80aee50 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 monclient: dump: 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.393 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbcb4003fd0 con 0x7fbcb80aee50 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcaffff640 4 set_mon_vals no callback set 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3445854125 0 0) 0x7fbcb4006870 con 0x7fbcb80aee50 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 10 client.4132.objecter ms_dispatch 0x7fbcb8000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 3 client.4132.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 3 client.4132.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.376+0000 7fbcae7fc640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.377+0000 7fbcc004e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.377+0000 7fbcc004e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.377+0000 7fbcc004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.377+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca0005080 con 0x7fbcb80aee50 2024-03-29T11:34:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.379+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbcb4004460 con 0x7fbcb80aee50 2024-03-29T11:34:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.379+0000 7fbcae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.379+0000 7fbcae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:40.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.451+0000 7fbcc004e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "057e8b01-3785-482b-8261-8b53a0386027", "id": 5}] 2024-03-29T11:34:40.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.451+0000 7fbcc004e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "057e8b01-3785-482b-8261-8b53a0386027", "id": 5}] 2024-03-29T11:34:40.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.451+0000 7fbcc004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.451+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "057e8b01-3785-482b-8261-8b53a0386027", "id": 5} v 0) v1 -- 0x7fbca00058b0 con 0x7fbcb80aee50 2024-03-29T11:34:40.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.455+0000 7fbcae7fc640 1 -- 172.21.15.153:0/1444342577 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "057e8b01-3785-482b-8261-8b53a0386027", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3072206910 0 1988412176) 0x7fbcb40063c0 con 0x7fbcb80aee50 2024-03-29T11:34:40.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.455+0000 7fbcae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "057e8b01-3785-482b-8261-8b53a0386027", "id": 5}] 2024-03-29T11:34:40.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.455+0000 7fbcae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:40.453 INFO:teuthology.orchestra.run.smithi153.stdout:5 2024-03-29T11:34:40.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 10 librados: watch_flush enter 2024-03-29T11:34:40.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 10 librados: watch_flush exit 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 10 monclient: shutdown 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 >> v1:172.21.15.153:6789/0 conn(0x7fbcb80aee50 legacy=0x7fbcb80b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 shutdown_connections 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 >> 172.21.15.153:0/1444342577 conn(0x7fbcb8066b00 msgr2=0x7fbcb813a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 shutdown_connections 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 1 -- 172.21.15.153:0/1444342577 wait complete. 2024-03-29T11:34:40.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.456+0000 7fbcc004e640 1 librados: shutdown 2024-03-29T11:34:40.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 17b518fe-d9df-4355-93b9-bc2f826b30cd 6 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.567+0000 7f417e332640 10 monclient: get_monmap_and_config 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.567+0000 7f417e332640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.567+0000 7f417e332640 10 monclient: monmap: 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:40.568652+0000 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:40.568652+0000 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:40.566 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:40.566 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.567+0000 7f417e332640 1 Processor -- start 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.567+0000 7f417e332640 1 -- start start 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.567+0000 7f417e332640 10 monclient: init 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 10 monclient(hunting): picked mon.noname-b con 0x7f417809ac40 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 10 monclient(hunting): picked mon.noname-a con 0x7f417809c870 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 10 monclient(hunting): picked mon.noname-c con 0x7f41780b4360 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f417809dec0 con 0x7f417809c870 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f417809da30 con 0x7f41780b4360 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780b39c0 con 0x7f417809ac40 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417e332640 10 monclient(hunting): authenticate will time out at 869.870305s 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417c8cf640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f41780b4360 0x7f417814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56286/0 (socket says 172.21.15.153:56286) 2024-03-29T11:34:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f417c8cf640 1 -- 172.21.15.153:0/2459873233 learned_addr learned my addr 172.21.15.153:0/2459873233 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835738545 0 0) 0x7f417809da30 con 0x7f41780b4360 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4160003520 con 0x7f41780b4360 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064088702 0 0) 0x7f41780b39c0 con 0x7f417809ac40 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f417809da30 con 0x7f417809ac40 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007004650 0 0) 0x7f416c001110 con 0x7f417809c870 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.568+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41780b39c0 con 0x7f417809c870 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665633083 0 0) 0x7f417809da30 con 0x7f417809ac40 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 >> v1:172.21.15.153:6789/0 conn(0x7f41780b4360 legacy=0x7f417814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 >> v1:172.21.15.33:6789/0 conn(0x7f417809c870 legacy=0x7f417813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 monclient: found mon.noname-b 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41780b5160 con 0x7f417809ac40 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4168001670 con 0x7f417809ac40 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: dump: 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.578 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 5 monclient: authenticate success, global_id 4148 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4168001850 con 0x7f417809ac40 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 1 -- 172.21.15.153:0/2459873233 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4168001ca0 con 0x7f417809ac40 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:40.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f4176ffd640 10 monclient: dump: 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 4 set_mon_vals no callback set 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 10 monclient: shutdown 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 1 -- 172.21.15.153:0/2459873233 >> v1:172.21.15.33:6790/0 conn(0x7f417809ac40 legacy=0x7f41780b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 1 -- 172.21.15.153:0/2459873233 shutdown_connections 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.569+0000 7f417e332640 1 -- 172.21.15.153:0/2459873233 >> 172.21.15.153:0/2459873233 conn(0x7f4178066b00 msgr2=0x7f4178066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.570+0000 7f417e332640 1 -- 172.21.15.153:0/2459873233 shutdown_connections 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.570+0000 7f417e332640 1 -- 172.21.15.153:0/2459873233 wait complete. 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.570+0000 7f417e332640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.570+0000 7f417e332640 10 monclient: monmap: 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:40.581 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.570+0000 7f417e332640 1 librados: starting msgr at 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.570+0000 7f417e332640 1 librados: starting objecter 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.570+0000 7f417e332640 1 Processor -- start 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 1 -- start start 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 1 librados: setting wanted keys 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 1 librados: calling monclient init 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient: init 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient(hunting): picked mon.noname-b con 0x7f417809ac40 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient(hunting): picked mon.noname-a con 0x7f41780b0ec0 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient(hunting): picked mon.noname-c con 0x7f41781461f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4178141070 con 0x7f41780b0ec0 2024-03-29T11:34:40.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780ae0e0 con 0x7f417809ac40 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780b1ae0 con 0x7f41781461f0 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f417e332640 10 monclient(hunting): authenticate will time out at 869.873594s 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f4177fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f417809ac40 0x7f41781d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56288/0 (socket says 172.21.15.153:56288) 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f4177fff640 1 -- 172.21.15.153:0/2429339902 learned_addr learned my addr 172.21.15.153:0/2429339902 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f4174ff9640 10 client.?.objecter ms_handle_connect 0x7f417809ac40 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f4174ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.571+0000 7f4174ff9640 10 client.?.objecter ms_handle_connect 0x7f41780b0ec0 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 client.?.objecter ms_handle_connect 0x7f41781461f0 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514170226 0 0) 0x7f41780ae0e0 con 0x7f417809ac40 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4164004eb0 con 0x7f417809ac40 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655253914 0 0) 0x7f4178141070 con 0x7f41780b0ec0 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41780ae0e0 con 0x7f41780b0ec0 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175455551 0 0) 0x7f41780b1ae0 con 0x7f41781461f0 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4178141070 con 0x7f41781461f0 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545456147 0 0) 0x7f4164004eb0 con 0x7f417809ac40 2024-03-29T11:34:40.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41780b1ae0 con 0x7f417809ac40 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4168001ca0 con 0x7f417809ac40 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:40.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2667775034 0 0) 0x7f41780ae0e0 con 0x7f41780b0ec0 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4164004eb0 con 0x7f41780b0ec0 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2099680641 0 0) 0x7f4178141070 con 0x7f41781461f0 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41780ae0e0 con 0x7f41781461f0 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f416c001860 con 0x7f41780b0ec0 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.586 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f41700019d0 con 0x7f41781461f0 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.587 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1788430692 0 0) 0x7f41780b1ae0 con 0x7f417809ac40 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 >> v1:172.21.15.33:6790/0 conn(0x7f41781461f0 legacy=0x7f41781e2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 >> v1:172.21.15.33:6789/0 conn(0x7f41780b0ec0 legacy=0x7f4178145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 monclient: found mon.b 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41780b0930 con 0x7f417809ac40 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 10 monclient: _renew_subs 2024-03-29T11:34:40.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417813ffa0 con 0x7f417809ac40 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 10 monclient: _renew_subs 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41781d8020 con 0x7f417809ac40 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f417e332640 1 librados: init done 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4168003960 con 0x7f417809ac40 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4168003d70 con 0x7f417809ac40 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 monclient: dump: 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.589 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f41767fc640 4 set_mon_vals no callback set 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f41680042e0 con 0x7f417809ac40 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3320491059 0 0) 0x7f4168006590 con 0x7f417809ac40 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 10 client.4138.objecter ms_dispatch 0x7f4178000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 3 client.4138.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:34:40.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.572+0000 7f4174ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:34:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.573+0000 7f417e332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.573+0000 7f417e332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.573+0000 7f417e332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.573+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4158005080 con 0x7f417809ac40 2024-03-29T11:34:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.575+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41680067a0 con 0x7f417809ac40 2024-03-29T11:34:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.575+0000 7f4174ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.575+0000 7f4174ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:40.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.648+0000 7f417e332640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "17b518fe-d9df-4355-93b9-bc2f826b30cd", "id": 6}] 2024-03-29T11:34:40.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.648+0000 7f417e332640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "17b518fe-d9df-4355-93b9-bc2f826b30cd", "id": 6}] 2024-03-29T11:34:40.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.648+0000 7f417e332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.648+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "17b518fe-d9df-4355-93b9-bc2f826b30cd", "id": 6} v 0) v1 -- 0x7f41580058b0 con 0x7f417809ac40 2024-03-29T11:34:40.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.652+0000 7f4174ff9640 1 -- 172.21.15.153:0/2429339902 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17b518fe-d9df-4355-93b9-bc2f826b30cd", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2510693017 0 1113811849) 0x7f4168006940 con 0x7f417809ac40 2024-03-29T11:34:40.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.652+0000 7f4174ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "17b518fe-d9df-4355-93b9-bc2f826b30cd", "id": 6}] 2024-03-29T11:34:40.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.652+0000 7f4174ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:40.651 INFO:teuthology.orchestra.run.smithi153.stdout:6 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 10 librados: watch_flush enter 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 10 librados: watch_flush exit 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 10 monclient: shutdown 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 >> v1:172.21.15.153:6789/0 conn(0x7f417809ac40 legacy=0x7f41781d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 shutdown_connections 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 >> 172.21.15.153:0/2429339902 conn(0x7f4178066b00 msgr2=0x7f417813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 shutdown_connections 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 1 -- 172.21.15.153:0/2429339902 wait complete. 2024-03-29T11:34:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.653+0000 7f417e332640 1 librados: shutdown 2024-03-29T11:34:40.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new b7280579-d574-43a9-a294-0b40c1b3b700 7 2024-03-29T11:34:40.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:40.724+0000 7fed89632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:40.726 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:40.729+0000 7f4f95610040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.764+0000 7fdbc943e640 10 monclient: get_monmap_and_config 2024-03-29T11:34:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.764+0000 7fdbc943e640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.765+0000 7fdbc943e640 10 monclient: monmap: 2024-03-29T11:34:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:40.766554+0000 2024-03-29T11:34:40.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:40.766554+0000 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.765+0000 7fdbc943e640 1 Processor -- start 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.765+0000 7fdbc943e640 1 -- start start 2024-03-29T11:34:40.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.765+0000 7fdbc943e640 10 monclient: init 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc409ac40 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc409c870 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc40b4360 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc409dec0 con 0x7fdbc40b4360 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc409da30 con 0x7fdbc409c870 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40b39c0 con 0x7fdbc409ac40 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient(hunting): authenticate will time out at 870.068039s 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc27fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdbc409c870 0x7fdbc413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56292/0 (socket says 172.21.15.153:56292) 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc27fc640 1 -- 172.21.15.153:0/1989975525 learned_addr learned my addr 172.21.15.153:0/1989975525 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464914333 0 0) 0x7fdbc409da30 con 0x7fdbc409c870 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbac003520 con 0x7fdbc409c870 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804859518 0 0) 0x7fdbc40b39c0 con 0x7fdbc409ac40 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc409da30 con 0x7fdbc409ac40 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140839502 0 0) 0x7fdbbc0013c0 con 0x7fdbc40b4360 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc40b39c0 con 0x7fdbc40b4360 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090490150 0 0) 0x7fdbac003520 con 0x7fdbc409c870 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 >> v1:172.21.15.33:6790/0 conn(0x7fdbc409ac40 legacy=0x7fdbc40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 >> v1:172.21.15.33:6789/0 conn(0x7fdbc40b4360 legacy=0x7fdbc414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 monclient: found mon.noname-c 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40b5160 con 0x7fdbc409c870 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdbb4001690 con 0x7fdbc409c870 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: dump: 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.776 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.777 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.777 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.777 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbb4001870 con 0x7fdbc409c870 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 1 -- 172.21.15.153:0/1989975525 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdbb4001c80 con 0x7fdbc409c870 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc943e640 4 set_mon_vals no callback set 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.766+0000 7fdbc1ffb640 10 monclient: dump: 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.767+0000 7fdbc943e640 10 monclient: shutdown 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.767+0000 7fdbc943e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.767+0000 7fdbc943e640 1 -- 172.21.15.153:0/1989975525 >> v1:172.21.15.153:6789/0 conn(0x7fdbc409c870 legacy=0x7fdbc413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.767+0000 7fdbc943e640 1 -- 172.21.15.153:0/1989975525 shutdown_connections 2024-03-29T11:34:40.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.767+0000 7fdbc943e640 1 -- 172.21.15.153:0/1989975525 >> 172.21.15.153:0/1989975525 conn(0x7fdbc4066b00 msgr2=0x7fdbc4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.767+0000 7fdbc943e640 1 -- 172.21.15.153:0/1989975525 shutdown_connections 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.767+0000 7fdbc943e640 1 -- 172.21.15.153:0/1989975525 wait complete. 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 10 monclient: monmap: 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 1 librados: starting msgr at 2024-03-29T11:34:40.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 1 librados: starting objecter 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 1 Processor -- start 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 1 -- start start 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 1 librados: setting wanted keys 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 1 librados: calling monclient init 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.768+0000 7fdbc943e640 10 monclient: init 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc409ac40 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc40afa60 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc41db5a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40af690 con 0x7fdbc40afa60 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc411fbb0 con 0x7fdbc409ac40 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40afef0 con 0x7fdbc41db5a0 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc943e640 10 monclient(hunting): authenticate will time out at 870.071162s 2024-03-29T11:34:40.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdbc409ac40 0x7fdbc4139490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:56302/0 (socket says 172.21.15.153:56302) 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbc2ffd640 1 -- 172.21.15.153:0/893775953 learned_addr learned my addr 172.21.15.153:0/893775953 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 client.?.objecter ms_handle_connect 0x7fdbc409ac40 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 client.?.objecter ms_handle_connect 0x7fdbc40afa60 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 client.?.objecter ms_handle_connect 0x7fdbc41db5a0 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537699245 0 0) 0x7fdbc411fbb0 con 0x7fdbc409ac40 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb0004eb0 con 0x7fdbc409ac40 2024-03-29T11:34:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270536360 0 0) 0x7fdbc40af690 con 0x7fdbc40afa60 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc411fbb0 con 0x7fdbc40afa60 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858554344 0 0) 0x7fdbc40afef0 con 0x7fdbc41db5a0 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc40af690 con 0x7fdbc41db5a0 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2610271157 0 0) 0x7fdbb0004eb0 con 0x7fdbc409ac40 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc40afef0 con 0x7fdbc409ac40 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdbb80019b0 con 0x7fdbc409ac40 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): dump: 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.769+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3587338212 0 0) 0x7fdbc411fbb0 con 0x7fdbc40afa60 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb0004eb0 con 0x7fdbc40afa60 2024-03-29T11:34:40.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdbb4001b80 con 0x7fdbc40afa60 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient(hunting): dump: 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.785 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441243294 0 0) 0x7fdbc40af690 con 0x7fdbc41db5a0 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc411fbb0 con 0x7fdbc41db5a0 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdbbc001a70 con 0x7fdbc41db5a0 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient(hunting): dump: 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.786 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3534277546 0 0) 0x7fdbc40afef0 con 0x7fdbc409ac40 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 >> v1:172.21.15.33:6790/0 conn(0x7fdbc41db5a0 legacy=0x7fdbc41e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 >> v1:172.21.15.33:6789/0 conn(0x7fdbc40afa60 legacy=0x7fdbc40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 monclient: found mon.b 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc41dab60 con 0x7fdbc409ac40 2024-03-29T11:34:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 10 monclient: _renew_subs 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc41dadd0 con 0x7fdbc409ac40 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 10 monclient: _renew_subs 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc4139d80 con 0x7fdbc409ac40 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc943e640 1 librados: init done 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbb8003610 con 0x7fdbc409ac40 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdbb80039a0 con 0x7fdbc409ac40 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbc17fa640 4 set_mon_vals no callback set 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 monclient: dump: 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdbb8003f10 con 0x7fdbc409ac40 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3365077430 0 0) 0x7fdbb80068b0 con 0x7fdbc409ac40 2024-03-29T11:34:40.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 10 client.4144.objecter ms_dispatch 0x7fdbc4000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 3 client.4144.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 3 client.4144.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.770+0000 7fdbab7fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.771+0000 7fdbc943e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.771+0000 7fdbc943e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.771+0000 7fdbc943e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.771+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba4005080 con 0x7fdbc409ac40 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.773+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdbb8004350 con 0x7fdbc409ac40 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.773+0000 7fdbab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.773+0000 7fdbab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:40.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.845+0000 7fdbc943e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b7280579-d574-43a9-a294-0b40c1b3b700", "id": 7}] 2024-03-29T11:34:40.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.845+0000 7fdbc943e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b7280579-d574-43a9-a294-0b40c1b3b700", "id": 7}] 2024-03-29T11:34:40.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.845+0000 7fdbc943e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:40.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.845+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b7280579-d574-43a9-a294-0b40c1b3b700", "id": 7} v 0) v1 -- 0x7fdba40058b0 con 0x7fdbc409ac40 2024-03-29T11:34:40.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.848+0000 7fdbab7fe640 1 -- 172.21.15.153:0/893775953 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7280579-d574-43a9-a294-0b40c1b3b700", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 763975730 0 1371665406) 0x7fdbb8006ae0 con 0x7fdbc409ac40 2024-03-29T11:34:40.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.848+0000 7fdbab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b7280579-d574-43a9-a294-0b40c1b3b700", "id": 7}] 2024-03-29T11:34:40.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.848+0000 7fdbab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:40.846 INFO:teuthology.orchestra.run.smithi153.stdout:7 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 10 librados: watch_flush enter 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 10 librados: watch_flush exit 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 10 monclient: shutdown 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 >> v1:172.21.15.153:6789/0 conn(0x7fdbc409ac40 legacy=0x7fdbc4139490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 shutdown_connections 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 >> 172.21.15.153:0/893775953 conn(0x7fdbc4066b00 msgr2=0x7fdbc40b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 shutdown_connections 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 1 -- 172.21.15.153:0/893775953 wait complete. 2024-03-29T11:34:40.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-29T11:34:40.849+0000 7fdbc943e640 1 librados: shutdown 2024-03-29T11:34:40.857 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:34:40.857 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:34:40.859 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:34:40.859 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:34:40.859 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:34:40.861 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:34:40.862 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:34:40.862 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:34:40.863 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:40.865+0000 7fed89632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:40.864 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:34:40.864 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:34:40.864 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:34:40.866 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:34:40.867 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:34:40.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:34:40.869 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:34:40.869 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:34:40.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:34:40.870 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:40.872+0000 7f4f95610040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:40.871 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:34:40.871 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:34:40.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:34:40.873 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:34:40.873 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:34:40.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:34:40.875 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:34:40.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:41.005 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:41.007+0000 7fed89632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:41.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.011+0000 7f505cd9d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:41.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.012+0000 7f505cd9d640 10 monclient: build_initial_monmap 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.012+0000 7f505cd9d640 10 monclient: monmap: 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:41.012591+0000 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:41.012591+0000 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:41.009 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.012+0000 7f505cd9d640 1 Processor -- start 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.012+0000 7f505cd9d640 1 -- start start 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.012+0000 7f505cd9d640 10 monclient: init 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f505809ac40 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f505809c870 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f50580b4360 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505809dec0 con 0x7f505809ac40 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505809da30 con 0x7f50580b4360 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580b39c0 con 0x7f505809c870 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505cd9d640 10 monclient(hunting): authenticate will time out at 837.277040s 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f5055d9b640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f505809c870 0x7f505813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:45852/0 (socket says 172.21.15.33:45852) 2024-03-29T11:34:41.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f5055d9b640 1 -- 172.21.15.33:0/882550468 learned_addr learned my addr 172.21.15.33:0/882550468 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925705178 0 0) 0x7f50580b39c0 con 0x7f505809c870 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5040003520 con 0x7f505809c870 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541738149 0 0) 0x7f504c001110 con 0x7f505809ac40 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50580b39c0 con 0x7f505809ac40 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414007657 0 0) 0x7f505809da30 con 0x7f50580b4360 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 10 monclient(hunting): my global_id is 4171 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.013+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c001110 con 0x7f50580b4360 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 65545045 0 0) 0x7f5040003520 con 0x7f505809c870 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 >> v1:172.21.15.153:6789/0 conn(0x7f50580b4360 legacy=0x7f505814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 >> v1:172.21.15.33:6789/0 conn(0x7f505809ac40 legacy=0x7f50580b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 monclient: found mon.noname-b 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50580b5160 con 0x7f505809c870 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: _finish_auth 0 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:41.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5048001690 con 0x7f505809c870 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: dump: 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.016 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50480017f0 con 0x7f505809c870 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 1 -- 172.21.15.33:0/882550468 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5048001c40 con 0x7f505809c870 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 4 set_mon_vals no callback set 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505559a640 10 monclient: dump: 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.017 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 10 monclient: shutdown 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 1 -- 172.21.15.33:0/882550468 >> v1:172.21.15.33:6790/0 conn(0x7f505809c870 legacy=0x7f505813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 1 -- 172.21.15.33:0/882550468 shutdown_connections 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 1 -- 172.21.15.33:0/882550468 >> 172.21.15.33:0/882550468 conn(0x7f5058066b00 msgr2=0x7f5058066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 1 -- 172.21.15.33:0/882550468 shutdown_connections 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.014+0000 7f505cd9d640 1 -- 172.21.15.33:0/882550468 wait complete. 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.015+0000 7f505cd9d640 10 monclient: build_initial_monmap 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.015+0000 7f505cd9d640 10 monclient: monmap: 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-29T11:34:41.018 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.015+0000 7f505cd9d640 1 librados: starting msgr at 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.015+0000 7f505cd9d640 1 librados: starting objecter 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.015+0000 7f505cd9d640 1 Processor -- start 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 1 -- start start 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 1 librados: setting wanted keys 2024-03-29T11:34:41.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 1 librados: calling monclient init 2024-03-29T11:34:41.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient: init 2024-03-29T11:34:41.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f505809ac40 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f50581d6df0 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f50580b1210 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5055d9b640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f50581d6df0 0x7f505813a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:58062/0 (socket says 172.21.15.33:58062) 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5055d9b640 1 -- 172.21.15.33:0/2085017738 learned_addr learned my addr 172.21.15.33:0/2085017738 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50581d7360 con 0x7f50581d6df0 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505813b800 con 0x7f505809ac40 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 client.?.objecter ms_handle_connect 0x7f50581d6df0 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580ad920 con 0x7f50580b1210 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:41.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f505cd9d640 10 monclient(hunting): authenticate will time out at 837.280504s 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 client.?.objecter ms_handle_connect 0x7f505809ac40 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 client.?.objecter ms_handle_connect 0x7f50580b1210 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839192844 0 0) 0x7f50581d7360 con 0x7f50581d6df0 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.016+0000 7f5046ffd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5038004eb0 con 0x7f50581d6df0 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116095300 0 0) 0x7f505813b800 con 0x7f505809ac40 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): my global_id is 4174 2024-03-29T11:34:41.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50581d7360 con 0x7f505809ac40 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249109408 0 0) 0x7f50580ad920 con 0x7f50580b1210 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f505813b800 con 0x7f50580b1210 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3430448023 0 0) 0x7f5038004eb0 con 0x7f50581d6df0 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50580ad920 con 0x7f50581d6df0 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5048001bc0 con 0x7f50581d6df0 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:41.026 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 124549315 0 0) 0x7f505813b800 con 0x7f50580b1210 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5038004eb0 con 0x7f50580b1210 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3138093844 0 0) 0x7f50581d7360 con 0x7f505809ac40 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f505813b800 con 0x7f505809ac40 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5050001b50 con 0x7f50580b1210 2024-03-29T11:34:41.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f504c0019c0 con 0x7f505809ac40 2024-03-29T11:34:41.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2128945658 0 0) 0x7f50580ad920 con 0x7f50581d6df0 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 >> v1:172.21.15.33:6790/0 conn(0x7f50580b1210 legacy=0x7f505813b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 >> v1:172.21.15.153:6789/0 conn(0x7f505809ac40 legacy=0x7f50581ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 monclient: found mon.a 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50581424e0 con 0x7f50581d6df0 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 10 monclient: _renew_subs 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5058142730 con 0x7f50581d6df0 2024-03-29T11:34:41.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 10 monclient: _renew_subs 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50581d79d0 con 0x7f50581d6df0 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f505cd9d640 1 librados: init done 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5048004bf0 con 0x7f50581d6df0 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5046ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.017+0000 7f5054d99640 4 set_mon_vals no callback set 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5048004f00 con 0x7f50581d6df0 2024-03-29T11:34:41.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.031 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f4f95610040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:41.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:41.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 10 monclient: dump: 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:34:41.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f50480053f0 con 0x7f50581d6df0 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f5048006b60 con 0x7f50581d6df0 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 10 client.4191.objecter ms_dispatch 0x7f5058000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f5046ffd640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f505cd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f505cd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f505cd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.018+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c005080 con 0x7f50581d6df0 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.020+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f50480064c0 con 0x7f50581d6df0 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.020+0000 7f5046ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:41.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.020+0000 7f5046ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:41.050 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:41.052+0000 7fed89632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:41.062 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:41.065+0000 7f4f95610040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:41.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.082+0000 7f505cd9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:41.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.082+0000 7f505cd9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:41.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.082+0000 7f505cd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:41.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.082+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f503c005330 con 0x7f50581d6df0 2024-03-29T11:34:41.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f5046ffd640 1 -- 172.21.15.33:0/2085017738 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f5048006770 con 0x7f50581d6df0 2024-03-29T11:34:41.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f5046ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:41.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f5046ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:41.080 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-29T11:34:41.080 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:41.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 10 librados: watch_flush enter 2024-03-29T11:34:41.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 10 librados: watch_flush exit 2024-03-29T11:34:41.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:34:41.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 10 monclient: shutdown 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 >> v1:172.21.15.33:6789/0 conn(0x7f50581d6df0 legacy=0x7f505813a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 shutdown_connections 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.083+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 >> 172.21.15.33:0/2085017738 conn(0x7f5058066b00 msgr2=0x7f505813bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.084+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 shutdown_connections 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.084+0000 7f505cd9d640 1 -- 172.21.15.33:0/2085017738 wait complete. 2024-03-29T11:34:41.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-29T11:34:41.084+0000 7f505cd9d640 1 librados: shutdown 2024-03-29T11:34:41.091 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:34:41.091 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:34:41.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:41.163+0000 7fed89632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:41.179 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:41.182+0000 7f4f95610040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:41.257 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:41.259+0000 7fed89632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:41.278 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:41.281+0000 7f4f95610040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:41.410 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:41.412+0000 7fed89632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:41.438 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:41.441+0000 7f4f95610040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:41.451 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:41.453+0000 7fed89632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:41.470 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-29T11:34:41.472+0000 7f0be110f640 -1 Falling back to public interface 2024-03-29T11:34:41.471 INFO:tasks.ceph.osd.3.smithi033.stderr:2024-03-29T11:34:41.473+0000 7fb4dbbbd640 -1 Falling back to public interface 2024-03-29T11:34:41.471 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-29T11:34:41.472+0000 7ff346c1b640 -1 Falling back to public interface 2024-03-29T11:34:41.471 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-29T11:34:41.472+0000 7f0be00be640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T11:34:41.471 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-29T11:34:41.472+0000 7ff345605640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-03-29T11:34:41.476 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-29T11:34:41.478+0000 7f16ca21b640 -1 Falling back to public interface 2024-03-29T11:34:41.478 INFO:tasks.ceph.osd.2.smithi033.stderr:2024-03-29T11:34:41.481+0000 7fd81a714640 -1 Falling back to public interface 2024-03-29T11:34:41.478 INFO:tasks.ceph.osd.1.smithi033.stderr:2024-03-29T11:34:41.481+0000 7f53a55ac640 -1 Falling back to public interface 2024-03-29T11:34:41.479 INFO:tasks.ceph.osd.1.smithi033.stderr:2024-03-29T11:34:41.482+0000 7f53a455b640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2024-03-29T11:34:41.483 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:41.486+0000 7f4f95610040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:41.483 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-29T11:34:41.486+0000 7f30b78c8640 -1 Falling back to public interface 2024-03-29T11:34:41.493 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-03-29T11:34:41.496+0000 7f20d9c19640 -1 Falling back to public interface 2024-03-29T11:34:41.609 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:41.611+0000 7fed89632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:41.643 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:41.646+0000 7f4f95610040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:41.896 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.988+0000 7f3f4e4f1640 10 monclient: get_monmap_and_config 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.988+0000 7f3f4e4f1640 10 monclient: build_initial_monmap 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.988+0000 7f3f4e4f1640 10 monclient: monmap: 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:41.989088+0000 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:41.989088+0000 2024-03-29T11:34:41.989 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 1 Processor -- start 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 1 -- start start 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient: init 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3f481457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:41.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4813f650 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3f480a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f480aaef0 con 0x7f3f4813f650 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f48145bd0 con 0x7f3f481457f0 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f48145eb0 con 0x7f3f480a9d70 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f4e4f1640 10 monclient(hunting): authenticate will time out at 838.253503s 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f477fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f3f4813f650 0x7f3f480b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:58070/0 (socket says 172.21.15.33:58070) 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.989+0000 7f3f477fe640 1 -- 172.21.15.33:0/3294261701 learned_addr learned my addr 172.21.15.33:0/3294261701 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524686782 0 0) 0x7f3f48145eb0 con 0x7f3f480a9d70 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f30003520 con 0x7f3f480a9d70 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237437941 0 0) 0x7f3f480aaef0 con 0x7f3f4813f650 2024-03-29T11:34:41.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f48145eb0 con 0x7f3f4813f650 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427125837 0 0) 0x7f3f48145bd0 con 0x7f3f481457f0 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f480aaef0 con 0x7f3f481457f0 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2501942856 0 0) 0x7f3f30003520 con 0x7f3f480a9d70 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 >> v1:172.21.15.153:6789/0 conn(0x7f3f481457f0 legacy=0x7f3f4813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 >> v1:172.21.15.33:6789/0 conn(0x7f3f4813f650 legacy=0x7f3f480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 monclient: found mon.noname-b 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:34:41.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48146340 con 0x7f3f480a9d70 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3f40001a10 con 0x7f3f480a9d70 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f4e4f1640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: dump: 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.993 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f4e4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f40001b70 con 0x7f3f480a9d70 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 1 -- 172.21.15.33:0/3294261701 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3f40001fc0 con 0x7f3f480a9d70 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f4e4f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f4e4f1640 4 set_mon_vals no callback set 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.990+0000 7f3f46ffd640 10 monclient: dump: 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:41.994 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 10 monclient: shutdown 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3294261701 >> v1:172.21.15.33:6790/0 conn(0x7f3f480a9d70 legacy=0x7f3f4814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3294261701 shutdown_connections 2024-03-29T11:34:41.995 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3294261701 >> 172.21.15.33:0/3294261701 conn(0x7f3f48066a00 msgr2=0x7f3f48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3294261701 shutdown_connections 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3294261701 wait complete. 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 10 monclient: build_initial_monmap 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.991+0000 7f3f4e4f1640 10 monclient: monmap: 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 librados: starting msgr at 2024-03-29T11:34:41.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 librados: starting objecter 2024-03-29T11:34:41.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 Processor -- start 2024-03-29T11:34:41.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 -- start start 2024-03-29T11:34:41.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 librados: setting wanted keys 2024-03-29T11:34:41.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 librados: calling monclient init 2024-03-29T11:34:41.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient: init 2024-03-29T11:34:42.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:42.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:42.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3f480a9d70 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:42.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4813f650 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:42.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3f481457f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:42.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f481377e0 con 0x7f3f4813f650 2024-03-29T11:34:42.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f480ac860 con 0x7f3f480a9d70 2024-03-29T11:34:42.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4813fe40 con 0x7f3f481457f0 2024-03-29T11:34:42.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:42.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4e4f1640 10 monclient(hunting): authenticate will time out at 838.256547s 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f477fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f3f4813f650 0x7f3f481369c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:58080/0 (socket says 172.21.15.33:58080) 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f477fe640 1 -- 172.21.15.33:0/3676593467 learned_addr learned my addr 172.21.15.33:0/3676593467 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.992+0000 7f3f4ca8e640 1 --1- 172.21.15.33:0/3676593467 >> v1:172.21.15.33:6790/0 conn(0x7f3f481457f0 0x7f3f481370d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:45880/0 (socket says 172.21.15.33:45880) 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 client.?.objecter ms_handle_connect 0x7f3f4813f650 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 client.?.objecter ms_handle_connect 0x7f3f481457f0 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 client.?.objecter ms_handle_connect 0x7f3f480a9d70 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451164329 0 0) 0x7f3f481377e0 con 0x7f3f4813f650 2024-03-29T11:34:42.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f34004eb0 con 0x7f3f4813f650 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042779107 0 0) 0x7f3f4813fe40 con 0x7f3f481457f0 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f481377e0 con 0x7f3f481457f0 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536411292 0 0) 0x7f3f480ac860 con 0x7f3f480a9d70 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f4813fe40 con 0x7f3f480a9d70 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 15579832 0 0) 0x7f3f34004eb0 con 0x7f3f4813f650 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f480ac860 con 0x7f3f4813f650 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3f3c001a50 con 0x7f3f4813f650 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:42.005 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2594810072 0 0) 0x7f3f481377e0 con 0x7f3f481457f0 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f34004eb0 con 0x7f3f481457f0 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3f40001e80 con 0x7f3f481457f0 2024-03-29T11:34:42.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:42.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 642738193 0 0) 0x7f3f4813fe40 con 0x7f3f480a9d70 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f481377e0 con 0x7f3f480a9d70 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3f38001790 con 0x7f3f480a9d70 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:42.008 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 493894909 0 0) 0x7f3f480ac860 con 0x7f3f4813f650 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 >> v1:172.21.15.33:6790/0 conn(0x7f3f481457f0 legacy=0x7f3f481370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 >> v1:172.21.15.153:6789/0 conn(0x7f3f480a9d70 legacy=0x7f3f48140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 monclient: found mon.a 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f480ac580 con 0x7f3f4813f650 2024-03-29T11:34:42.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 5 monclient: authenticate success, global_id 4197 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 10 monclient: _renew_subs 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f480ab600 con 0x7f3f4813f650 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 10 monclient: _renew_subs 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f48141b80 con 0x7f3f4813f650 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f4e4f1640 1 librados: init done 2024-03-29T11:34:42.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f3c001c50 con 0x7f3f4813f650 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3f3c004c30 con 0x7f3f4813f650 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f44ff9640 10 monclient: dump: 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:42.011 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.993+0000 7f3f467fc640 4 set_mon_vals no callback set 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3f3c005120 con 0x7f3f4813f650 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f3f3c006980 con 0x7f3f4813f650 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f44ff9640 10 client.4197.objecter ms_dispatch 0x7f3f48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f44ff9640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f44ff9640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f44ff9640 20 client.4197.objecter dump_active .. 0 homeless 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f4e4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f4e4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f4e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.994+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f48075f10 con 0x7f3f4813f650 2024-03-29T11:34:42.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.998+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3f3c0063c0 con 0x7f3f4813f650 2024-03-29T11:34:42.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.998+0000 7f3f44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:41.998+0000 7f3f44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:42.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.061+0000 7f3f4e4f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.061+0000 7f3f4e4f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.061+0000 7f3f4e4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:42.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.061+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f480b3680 con 0x7f3f4813f650 2024-03-29T11:34:42.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f44ff9640 1 -- 172.21.15.33:0/3676593467 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f3f3c006610 con 0x7f3f4813f650 2024-03-29T11:34:42.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:42.059 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:34:42.059 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f4e4f1640 10 librados: watch_flush enter 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f4e4f1640 10 librados: watch_flush exit 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f4e4f1640 20 client.4197.objecter shutdown clearing up homeless session... 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f4e4f1640 10 client.4197.objecter successfully canceled tick 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.062+0000 7f3f4e4f1640 10 monclient: shutdown 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.063+0000 7f3f4e4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.063+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 >> v1:172.21.15.33:6789/0 conn(0x7f3f4813f650 legacy=0x7f3f481369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.063+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 shutdown_connections 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.063+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 >> 172.21.15.33:0/3676593467 conn(0x7f3f48066a00 msgr2=0x7f3f480b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.063+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 shutdown_connections 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.063+0000 7f3f4e4f1640 1 -- 172.21.15.33:0/3676593467 wait complete. 2024-03-29T11:34:42.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:42.063+0000 7f3f4e4f1640 1 librados: shutdown 2024-03-29T11:34:42.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:42.449 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:42.441+0000 7fed89632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:42.496 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:42.498+0000 7f4f95610040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:42.716 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:42.719+0000 7fed89632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:42.773 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:42.769+0000 7fed89632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:42.781 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:42.784+0000 7f4f95610040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:42.835 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:42.837+0000 7f4f95610040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:42.883 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:42.886+0000 7fed89632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:42.958 INFO:tasks.ceph.mgr.y.smithi033.stderr:2024-03-29T11:34:42.959+0000 7f4f95610040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:43.897 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:43.900+0000 7fed29a7d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:34:43.898 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-29T11:34:43.901+0000 7fed29a7d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:34:44.935 INFO:tasks.ceph.osd.3.smithi033.stderr:2024-03-29T11:34:44.938+0000 7fb4dbbbd640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:34:44.955 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-29T11:34:44.958+0000 7ff346c1b640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:34:45.007 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-29T11:34:45.010+0000 7f0be110f640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:34:45.008 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-29T11:34:45.011+0000 7f30b78c8640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:34:45.008 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-29T11:34:45.011+0000 7f16ca21b640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:34:45.013 INFO:tasks.ceph.osd.0.smithi033.stderr:2024-03-29T11:34:45.016+0000 7f20d9c19640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:34:45.028 INFO:tasks.ceph.osd.1.smithi033.stderr:2024-03-29T11:34:45.031+0000 7f53a55ac640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:34:45.030 INFO:tasks.ceph.osd.2.smithi033.stderr:2024-03-29T11:34:45.032+0000 7fd81a714640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:34:48.875 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:49.050 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.052+0000 7f5515b7c640 10 monclient: get_monmap_and_config 2024-03-29T11:34:49.050 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.052+0000 7f5515b7c640 10 monclient: build_initial_monmap 2024-03-29T11:34:49.050 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.052+0000 7f5515b7c640 10 monclient: monmap: 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:49.053406+0000 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:49.053406+0000 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:49.051 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.054+0000 7f5515b7c640 1 Processor -- start 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.054+0000 7f5515b7c640 1 -- start start 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.054+0000 7f5515b7c640 10 monclient: init 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.055+0000 7f5515b7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.055+0000 7f5515b7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.055+0000 7f5515b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f55101457f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.055+0000 7f5515b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f551013f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.055+0000 7f5515b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f55100a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:49.054 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.056+0000 7f5515b7c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55100aaef0 con 0x7f55100a9d70 2024-03-29T11:34:49.054 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.056+0000 7f5515b7c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5510145bd0 con 0x7f551013f650 2024-03-29T11:34:49.054 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.056+0000 7f5515b7c640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5510145eb0 con 0x7f55101457f0 2024-03-29T11:34:49.054 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.056+0000 7f5515b7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:49.054 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.056+0000 7f5515b7c640 10 monclient(hunting): authenticate will time out at 845.319829s 2024-03-29T11:34:49.054 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.056+0000 7f550f7fe640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f55101457f0 0x7f551013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:43026/0 (socket says 172.21.15.33:43026) 2024-03-29T11:34:49.054 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.056+0000 7f550f7fe640 1 -- 172.21.15.33:0/4098085820 learned_addr learned my addr 172.21.15.33:0/4098085820 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:49.056 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173618926 0 0) 0x7f5510145bd0 con 0x7f551013f650 2024-03-29T11:34:49.056 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.056 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): my global_id is 4222 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54fc003520 con 0x7f551013f650 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792884557 0 0) 0x7f5510145eb0 con 0x7f55101457f0 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5510145bd0 con 0x7f55101457f0 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013163680 0 0) 0x7f54fc003520 con 0x7f551013f650 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 >> v1:172.21.15.33:6790/0 conn(0x7f55101457f0 legacy=0x7f551013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 >> v1:172.21.15.33:6789/0 conn(0x7f55100a9d70 legacy=0x7f551014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 monclient: found mon.noname-c 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5510146340 con 0x7f551013f650 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient: _finish_auth 0 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f54f8001690 con 0x7f551013f650 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f550effd640 10 monclient: dump: 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:49.058 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f5515b7c640 5 monclient: authenticate success, global_id 4222 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.057+0000 7f5515b7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54f8001870 con 0x7f551013f650 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f550effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f550effd640 1 -- 172.21.15.33:0/4098085820 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f54f8001c80 con 0x7f551013f650 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f550effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f550effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:49.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f550effd640 10 monclient: dump: 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f5515b7c640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:49.060 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.059+0000 7f5515b7c640 4 set_mon_vals no callback set 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.060+0000 7f5515b7c640 10 monclient: shutdown 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.060+0000 7f5515b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.060+0000 7f5515b7c640 1 -- 172.21.15.33:0/4098085820 >> v1:172.21.15.153:6789/0 conn(0x7f551013f650 legacy=0x7f55100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.060+0000 7f5515b7c640 1 -- 172.21.15.33:0/4098085820 shutdown_connections 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.060+0000 7f5515b7c640 1 -- 172.21.15.33:0/4098085820 >> 172.21.15.33:0/4098085820 conn(0x7f5510066a00 msgr2=0x7f5510066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.060+0000 7f5515b7c640 1 -- 172.21.15.33:0/4098085820 shutdown_connections 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.060+0000 7f5515b7c640 1 -- 172.21.15.33:0/4098085820 wait complete. 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.062+0000 7f5515b7c640 10 monclient: build_initial_monmap 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.062+0000 7f5515b7c640 10 monclient: monmap: 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:34:49.069 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.062+0000 7f5515b7c640 1 librados: starting msgr at 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.062+0000 7f5515b7c640 1 librados: starting objecter 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.063+0000 7f5515b7c640 1 Processor -- start 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.063+0000 7f5515b7c640 1 -- start start 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.063+0000 7f5515b7c640 1 librados: setting wanted keys 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.063+0000 7f5515b7c640 1 librados: calling monclient init 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.063+0000 7f5515b7c640 10 monclient: init 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:49.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f55100a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f551013f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f55101457f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55101387a0 con 0x7f55100a9d70 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551006f600 con 0x7f551013f650 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551006f8e0 con 0x7f55101457f0 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5515b7c640 10 monclient(hunting): authenticate will time out at 845.327817s 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5507fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f551013f650 0x7f5510137980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:38214/0 (socket says 172.21.15.33:38214) 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f5507fff640 1 -- 172.21.15.33:0/2442077561 learned_addr learned my addr 172.21.15.33:0/2442077561 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f550cff9640 10 client.?.objecter ms_handle_connect 0x7f551013f650 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f550cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f550cff9640 10 client.?.objecter ms_handle_connect 0x7f55100a9d70 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f550cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f550cff9640 10 client.?.objecter ms_handle_connect 0x7f55101457f0 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.064+0000 7f550cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997043608 0 0) 0x7f55101387a0 con 0x7f55100a9d70 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f0004eb0 con 0x7f55100a9d70 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404638905 0 0) 0x7f551006f600 con 0x7f551013f650 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): my global_id is 4225 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55101387a0 con 0x7f551013f650 2024-03-29T11:34:49.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000259297 0 0) 0x7f551006f8e0 con 0x7f55101457f0 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f551006f600 con 0x7f55101457f0 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.065+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961790620 0 0) 0x7f55101387a0 con 0x7f551013f650 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f551006f8e0 con 0x7f551013f650 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f54f8001c80 con 0x7f551013f650 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient(hunting): dump: 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.073 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3320867527 0 0) 0x7f551006f8e0 con 0x7f551013f650 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 >> v1:172.21.15.33:6790/0 conn(0x7f55101457f0 legacy=0x7f5510138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 >> v1:172.21.15.33:6789/0 conn(0x7f55100a9d70 legacy=0x7f55101dbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 1 monclient: found mon.b 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:49.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55101db8d0 con 0x7f551013f650 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f550cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f5515b7c640 5 monclient: authenticate success, global_id 4225 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f5515b7c640 10 monclient: _renew_subs 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f5515b7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.066+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55101dc500 con 0x7f551013f650 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f5515b7c640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f5515b7c640 10 monclient: _renew_subs 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f5515b7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5510138ea0 con 0x7f551013f650 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f5515b7c640 1 librados: init done 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54f8003940 con 0x7f551013f650 2024-03-29T11:34:49.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f550cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f54f8003cd0 con 0x7f551013f650 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f550cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f550cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f550e7fc640 4 set_mon_vals no callback set 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.067+0000 7f550cff9640 10 monclient: dump: 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:49.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.068+0000 7f5515b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.068+0000 7f5515b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.068+0000 7f5515b7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.068+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f4005080 con 0x7f551013f650 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.070+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f54f8003e30 con 0x7f551013f650 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.070+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f54f8054960 con 0x7f551013f650 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.070+0000 7f550cff9640 10 client.4225.objecter ms_dispatch 0x7f5510000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.070+0000 7f550cff9640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.070+0000 7f550cff9640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.070+0000 7f550cff9640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.071+0000 7f550f7fe640 10 monclient: get_auth_request con 0x7f54f006de00 auth_method 0 2024-03-29T11:34:49.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.071+0000 7f550cff9640 10 client.4225.objecter ms_handle_connect 0x7f54f006de00 2024-03-29T11:34:49.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.077+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54f8050c00 con 0x7f551013f650 2024-03-29T11:34:49.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.077+0000 7f550cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.077+0000 7f550cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:49.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.457+0000 7f5515b7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:49.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.457+0000 7f5515b7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:49.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.457+0000 7f5515b7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:34:49.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.457+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54f4005330 con 0x7f551013f650 2024-03-29T11:34:49.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.458+0000 7f550cff9640 1 -- 172.21.15.33:0/2442077561 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f54f8050da0 con 0x7f551013f650 2024-03-29T11:34:49.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.458+0000 7f550cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:49.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.458+0000 7f550cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:49.456 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:34:49.456 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:49.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.461+0000 7f5515b7c640 10 librados: watch_flush enter 2024-03-29T11:34:49.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 10 librados: watch_flush exit 2024-03-29T11:34:49.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 10 client.4225.objecter successfully canceled tick 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 >> v1:172.21.15.153:6816/2737028074 conn(0x7f54f006de00 legacy=0x7f54f00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 10 monclient: shutdown 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 >> v1:172.21.15.153:6789/0 conn(0x7f551013f650 legacy=0x7f5510137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 shutdown_connections 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 >> 172.21.15.33:0/2442077561 conn(0x7f5510066a00 msgr2=0x7f551013cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 shutdown_connections 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 1 -- 172.21.15.33:0/2442077561 wait complete. 2024-03-29T11:34:49.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:49.462+0000 7f5515b7c640 1 librados: shutdown 2024-03-29T11:34:49.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:56.280 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.434+0000 7f72db5ed640 10 monclient: get_monmap_and_config 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.434+0000 7f72db5ed640 10 monclient: build_initial_monmap 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.434+0000 7f72db5ed640 10 monclient: monmap: 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:56.435283+0000 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:56.435283+0000 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:56.432 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:34:56.433 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:34:56.433 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:56.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.435+0000 7f72db5ed640 1 Processor -- start 2024-03-29T11:34:56.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.436+0000 7f72db5ed640 1 -- start start 2024-03-29T11:34:56.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.436+0000 7f72db5ed640 10 monclient: init 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f72d40b46b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f72d40aeae0 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f72d413fa10 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d40b06a0 con 0x7f72d40aeae0 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d40b4230 con 0x7f72d413fa10 2024-03-29T11:34:56.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d40b0910 con 0x7f72d40b46b0 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72db5ed640 10 monclient(hunting): authenticate will time out at 852.701302s 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72d9389640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f72d40b46b0 0x7f72d40b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:52974/0 (socket says 172.21.15.33:52974) 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72d9389640 1 -- 172.21.15.33:0/791274072 learned_addr learned my addr 172.21.15.33:0/791274072 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.437+0000 7f72d8b88640 1 --1- 172.21.15.33:0/791274072 >> v1:172.21.15.33:6789/0 conn(0x7f72d40aeae0 0x7f72d413f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49068/0 (socket says 172.21.15.33:49068) 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052494126 0 0) 0x7f72d40b06a0 con 0x7f72d40aeae0 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c0003520 con 0x7f72d40aeae0 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059558421 0 0) 0x7f72d40b0910 con 0x7f72d40b46b0 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d40b06a0 con 0x7f72d40b46b0 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175278087 0 0) 0x7f72c0003520 con 0x7f72d40aeae0 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 >> v1:172.21.15.33:6790/0 conn(0x7f72d40b46b0 legacy=0x7f72d40b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 >> v1:172.21.15.153:6789/0 conn(0x7f72d413fa10 legacy=0x7f72d414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 monclient: found mon.noname-a 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.438+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d40af6d0 con 0x7f72d40aeae0 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f72c4001540 con 0x7f72d40aeae0 2024-03-29T11:34:56.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: dump: 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72c4001760 con 0x7f72d40aeae0 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 1 -- 172.21.15.33:0/791274072 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f72c4001b30 con 0x7f72d40aeae0 2024-03-29T11:34:56.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72cbfff640 10 monclient: dump: 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72db5ed640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72db5ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:56.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.439+0000 7f72db5ed640 4 set_mon_vals no callback set 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.440+0000 7f72db5ed640 10 monclient: shutdown 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.440+0000 7f72db5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.440+0000 7f72db5ed640 1 -- 172.21.15.33:0/791274072 >> v1:172.21.15.33:6789/0 conn(0x7f72d40aeae0 legacy=0x7f72d413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.440+0000 7f72db5ed640 1 -- 172.21.15.33:0/791274072 shutdown_connections 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.440+0000 7f72db5ed640 1 -- 172.21.15.33:0/791274072 >> 172.21.15.33:0/791274072 conn(0x7f72d4066a00 msgr2=0x7f72d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.440+0000 7f72db5ed640 1 -- 172.21.15.33:0/791274072 shutdown_connections 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.441+0000 7f72db5ed640 1 -- 172.21.15.33:0/791274072 wait complete. 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.441+0000 7f72db5ed640 10 monclient: build_initial_monmap 2024-03-29T11:34:56.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.441+0000 7f72db5ed640 10 monclient: monmap: 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.442+0000 7f72db5ed640 1 librados: starting msgr at 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.442+0000 7f72db5ed640 1 librados: starting objecter 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.442+0000 7f72db5ed640 1 Processor -- start 2024-03-29T11:34:56.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.442+0000 7f72db5ed640 1 -- start start 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.442+0000 7f72db5ed640 1 librados: setting wanted keys 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.442+0000 7f72db5ed640 1 librados: calling monclient init 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.442+0000 7f72db5ed640 10 monclient: init 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f72d40b46b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f72d413fa10 addr v1:172.21.15.33:6789/0 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f72d413bd70 addr v1:172.21.15.153:6789/0 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d41356d0 con 0x7f72d413fa10 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d4146550 con 0x7f72d413bd70 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d4146270 con 0x7f72d40b46b0 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.443+0000 7f72db5ed640 10 monclient(hunting): authenticate will time out at 852.707442s 2024-03-29T11:34:56.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72d8b88640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f72d413fa10 0x7f72d41448d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49862/0 (socket says 172.21.15.33:49862) 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72d8b88640 1 -- 172.21.15.33:0/1828195680 learned_addr learned my addr 172.21.15.33:0/1828195680 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 client.?.objecter ms_handle_connect 0x7f72d413fa10 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 client.?.objecter ms_handle_connect 0x7f72d40b46b0 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341956534 0 0) 0x7f72d41356d0 con 0x7f72d413fa10 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72b8004eb0 con 0x7f72d413fa10 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821352868 0 0) 0x7f72d4146270 con 0x7f72d40b46b0 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:56.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.444+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d41356d0 con 0x7f72d40b46b0 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 client.?.objecter ms_handle_connect 0x7f72d413bd70 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2531158469 0 0) 0x7f72b8004eb0 con 0x7f72d413fa10 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72d4146270 con 0x7f72d413fa10 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f72c4001a70 con 0x7f72d413fa10 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:56.448 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2882036890 0 0) 0x7f72d4146270 con 0x7f72d413fa10 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 >> v1:172.21.15.33:6790/0 conn(0x7f72d40b46b0 legacy=0x7f72d4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 >> v1:172.21.15.153:6789/0 conn(0x7f72d413bd70 legacy=0x7f72d41e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 monclient: found mon.a 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:56.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d4144520 con 0x7f72d413fa10 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72db5ed640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72db5ed640 10 monclient: _renew_subs 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72db5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.445+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d4140ef0 con 0x7f72d413fa10 2024-03-29T11:34:56.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72db5ed640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:56.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72db5ed640 10 monclient: _renew_subs 2024-03-29T11:34:56.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72db5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:56.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d4141ad0 con 0x7f72d413fa10 2024-03-29T11:34:56.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72db5ed640 1 librados: init done 2024-03-29T11:34:56.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72c4004a90 con 0x7f72d413fa10 2024-03-29T11:34:56.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:56.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.446+0000 7f72cb7fe640 4 set_mon_vals no callback set 2024-03-29T11:34:56.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.447+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f72c4001180 con 0x7f72d413fa10 2024-03-29T11:34:56.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.447+0000 7f72c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:56.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.447+0000 7f72c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:56.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.447+0000 7f72c9ffb640 10 monclient: dump: 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.448+0000 7f72db5ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.448+0000 7f72db5ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:56.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.448+0000 7f72db5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.448+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b4005080 con 0x7f72d413fa10 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.449+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f72c401ac10 con 0x7f72d413fa10 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.449+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f72c4084ab0 con 0x7f72d413fa10 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.449+0000 7f72c9ffb640 10 client.4251.objecter ms_dispatch 0x7f72d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.449+0000 7f72c9ffb640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.449+0000 7f72c9ffb640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.450+0000 7f72c9ffb640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.450+0000 7f72d9389640 10 monclient: get_auth_request con 0x7f72b806dce0 auth_method 0 2024-03-29T11:34:56.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.450+0000 7f72c9ffb640 10 client.4251.objecter ms_handle_connect 0x7f72b806dce0 2024-03-29T11:34:56.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.457+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72c4050390 con 0x7f72d413fa10 2024-03-29T11:34:56.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.457+0000 7f72c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:56.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.457+0000 7f72c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.851+0000 7f72db5ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.851+0000 7f72db5ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.851+0000 7f72db5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.851+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72b4005330 con 0x7f72d413fa10 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.852+0000 7f72c9ffb640 1 -- 172.21.15.33:0/1828195680 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f72c40522c0 con 0x7f72d413fa10 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.852+0000 7f72c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.852+0000 7f72c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:56.851 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:34:56.852 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 10 librados: watch_flush enter 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 10 librados: watch_flush exit 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 >> v1:172.21.15.153:6816/2737028074 conn(0x7f72b806dce0 legacy=0x7f72b80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 10 monclient: shutdown 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.856+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 >> v1:172.21.15.33:6789/0 conn(0x7f72d413fa10 legacy=0x7f72d41448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.857+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 shutdown_connections 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.857+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 >> 172.21.15.33:0/1828195680 conn(0x7f72d4066a00 msgr2=0x7f72d40aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.857+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 shutdown_connections 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.857+0000 7f72db5ed640 1 -- 172.21.15.33:0/1828195680 wait complete. 2024-03-29T11:34:56.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:34:56.857+0000 7f72db5ed640 1 librados: shutdown 2024-03-29T11:34:56.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:03.674 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.819+0000 7fdc4cbcd640 10 monclient: get_monmap_and_config 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.819+0000 7fdc4cbcd640 10 monclient: build_initial_monmap 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.819+0000 7fdc4cbcd640 10 monclient: monmap: 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:03.820374+0000 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:03.820374+0000 2024-03-29T11:35:03.817 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.821+0000 7fdc4cbcd640 1 Processor -- start 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.821+0000 7fdc4cbcd640 1 -- start start 2024-03-29T11:35:03.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.821+0000 7fdc4cbcd640 10 monclient: init 2024-03-29T11:35:03.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:03.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:03.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 10 monclient(hunting): picked mon.noname-a con 0x7fdc481457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:03.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 10 monclient(hunting): picked mon.noname-c con 0x7fdc4813f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:03.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 10 monclient(hunting): picked mon.noname-b con 0x7fdc480a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:03.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480aaef0 con 0x7fdc481457f0 2024-03-29T11:35:03.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48145bd0 con 0x7fdc4813f650 2024-03-29T11:35:03.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48145eb0 con 0x7fdc480a9d70 2024-03-29T11:35:03.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:03.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.822+0000 7fdc4cbcd640 10 monclient(hunting): authenticate will time out at 860.085777s 2024-03-29T11:35:03.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.823+0000 7fdc45d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdc4813f650 0x7fdc480b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:41310/0 (socket says 172.21.15.33:41310) 2024-03-29T11:35:03.822 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.823+0000 7fdc45d9b640 1 -- 172.21.15.33:0/1462930708 learned_addr learned my addr 172.21.15.33:0/1462930708 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:03.822 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.824+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595728928 0 0) 0x7fdc48145bd0 con 0x7fdc4813f650 2024-03-29T11:35:03.822 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.824+0000 7fdc4559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.822 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.824+0000 7fdc4559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.822 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.824+0000 7fdc4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.822 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.824+0000 7fdc4559a640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:35:03.822 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.824+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc30003520 con 0x7fdc4813f650 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882270818 0 0) 0x7fdc48145eb0 con 0x7fdc480a9d70 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc48145bd0 con 0x7fdc480a9d70 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 300950018 0 0) 0x7fdc48145bd0 con 0x7fdc480a9d70 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 >> v1:172.21.15.153:6789/0 conn(0x7fdc4813f650 legacy=0x7fdc480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 >> v1:172.21.15.33:6789/0 conn(0x7fdc481457f0 legacy=0x7fdc4813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 monclient: found mon.noname-b 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48146340 con 0x7fdc480a9d70 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: _finish_auth 0 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc38001630 con 0x7fdc480a9d70 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: dump: 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:03.824 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4cbcd640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4cbcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc38001800 con 0x7fdc480a9d70 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4cbcd640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:03.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.825+0000 7fdc4cbcd640 4 set_mon_vals no callback set 2024-03-29T11:35:03.826 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.826+0000 7fdc4cbcd640 10 monclient: shutdown 2024-03-29T11:35:03.826 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.826+0000 7fdc4cbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:03.826 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.826+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/1462930708 >> v1:172.21.15.33:6790/0 conn(0x7fdc480a9d70 legacy=0x7fdc4814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.826 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.826+0000 7fdc4559a640 1 -- 172.21.15.33:0/1462930708 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc38001b90 con 0x7fdc480a9d70 2024-03-29T11:35:03.826 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.826+0000 7fdc4559a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:35:03.826 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.828+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/1462930708 shutdown_connections 2024-03-29T11:35:03.826 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.828+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/1462930708 >> 172.21.15.33:0/1462930708 conn(0x7fdc48066a00 msgr2=0x7fdc48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:03.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.829+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/1462930708 shutdown_connections 2024-03-29T11:35:03.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.829+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/1462930708 wait complete. 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.831+0000 7fdc4cbcd640 10 monclient: build_initial_monmap 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.831+0000 7fdc4cbcd640 10 monclient: monmap: 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:03.829 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:03.830 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:03.830 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:03.830 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:03.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.831+0000 7fdc4cbcd640 1 librados: starting msgr at 2024-03-29T11:35:03.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.831+0000 7fdc4cbcd640 1 librados: starting objecter 2024-03-29T11:35:03.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.831+0000 7fdc4cbcd640 1 Processor -- start 2024-03-29T11:35:03.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.832+0000 7fdc4cbcd640 1 -- start start 2024-03-29T11:35:03.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.832+0000 7fdc4cbcd640 1 librados: setting wanted keys 2024-03-29T11:35:03.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.832+0000 7fdc4cbcd640 1 librados: calling monclient init 2024-03-29T11:35:03.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.832+0000 7fdc4cbcd640 10 monclient: init 2024-03-29T11:35:03.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.833+0000 7fdc4cbcd640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:03.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.833+0000 7fdc4cbcd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:03.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.833+0000 7fdc4cbcd640 10 monclient(hunting): picked mon.noname-c con 0x7fdc480a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:03.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.833+0000 7fdc4cbcd640 10 monclient(hunting): picked mon.noname-a con 0x7fdc4813f650 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:03.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.833+0000 7fdc4cbcd640 10 monclient(hunting): picked mon.noname-b con 0x7fdc481457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:03.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.833+0000 7fdc4cbcd640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48137f20 con 0x7fdc4813f650 2024-03-29T11:35:03.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.833+0000 7fdc4cbcd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48138cc0 con 0x7fdc481457f0 2024-03-29T11:35:03.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.834+0000 7fdc4cbcd640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4813a2b0 con 0x7fdc480a9d70 2024-03-29T11:35:03.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.835+0000 7fdc4cbcd640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:03.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.835+0000 7fdc4cbcd640 10 monclient(hunting): authenticate will time out at 860.099507s 2024-03-29T11:35:03.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.834+0000 7fdc46d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdc481457f0 0x7fdc48137810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:41480/0 (socket says 172.21.15.33:41480) 2024-03-29T11:35:03.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc46d9d640 1 -- 172.21.15.33:0/2074789602 learned_addr learned my addr 172.21.15.33:0/2074789602 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:03.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc36ffd640 10 client.?.objecter ms_handle_connect 0x7fdc480a9d70 2024-03-29T11:35:03.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc36ffd640 10 client.?.objecter ms_handle_connect 0x7fdc481457f0 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc36ffd640 10 client.?.objecter ms_handle_connect 0x7fdc4813f650 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.837+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410947646 0 0) 0x7fdc4813a2b0 con 0x7fdc480a9d70 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:35:03.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc28004eb0 con 0x7fdc480a9d70 2024-03-29T11:35:03.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331294023 0 0) 0x7fdc48138cc0 con 0x7fdc481457f0 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.838+0000 7fdc36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc4813a2b0 con 0x7fdc481457f0 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 842477198 0 0) 0x7fdc28004eb0 con 0x7fdc480a9d70 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc48138cc0 con 0x7fdc480a9d70 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc40001c10 con 0x7fdc480a9d70 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:03.840 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.839+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3625779641 0 0) 0x7fdc48138cc0 con 0x7fdc480a9d70 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 >> v1:172.21.15.153:6789/0 conn(0x7fdc481457f0 legacy=0x7fdc48137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 >> v1:172.21.15.33:6789/0 conn(0x7fdc4813f650 legacy=0x7fdc48137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 1 monclient: found mon.c 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:03.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc481391c0 con 0x7fdc480a9d70 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 10 monclient: _renew_subs 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc48141f50 con 0x7fdc480a9d70 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 10 monclient: _renew_subs 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48142c40 con 0x7fdc480a9d70 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.840+0000 7fdc4cbcd640 1 librados: init done 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.841+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc40003970 con 0x7fdc480a9d70 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.841+0000 7fdc36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.841+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc40002100 con 0x7fdc480a9d70 2024-03-29T11:35:03.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.841+0000 7fdc44d99640 4 set_mon_vals no callback set 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.841+0000 7fdc36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.841+0000 7fdc36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.841+0000 7fdc36ffd640 10 monclient: dump: 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:03.843 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:03.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.843+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fdc40003d40 con 0x7fdc480a9d70 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fdc40053960 con 0x7fdc480a9d70 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc36ffd640 10 client.4253.objecter ms_dispatch 0x7fdc48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc36ffd640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc36ffd640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc36ffd640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc4cbcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc4cbcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc4cbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:03.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.845+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c005080 con 0x7fdc480a9d70 2024-03-29T11:35:03.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.848+0000 7fdc45d9b640 10 monclient: get_auth_request con 0x7fdc2806dbc0 auth_method 0 2024-03-29T11:35:03.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.848+0000 7fdc36ffd640 10 client.4253.objecter ms_handle_connect 0x7fdc2806dbc0 2024-03-29T11:35:03.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.853+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc4004f2a0 con 0x7fdc480a9d70 2024-03-29T11:35:03.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.853+0000 7fdc36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:03.853+0000 7fdc36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:04.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.187+0000 7fdc4cbcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.185 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:04.185 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.187+0000 7fdc4cbcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.187+0000 7fdc4cbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.187+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc2c005330 con 0x7fdc480a9d70 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.188+0000 7fdc36ffd640 1 -- 172.21.15.33:0/2074789602 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fdc400513a0 con 0x7fdc480a9d70 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.188+0000 7fdc36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.188+0000 7fdc36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.191+0000 7fdc4cbcd640 10 librados: watch_flush enter 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.191+0000 7fdc4cbcd640 10 librados: watch_flush exit 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.191+0000 7fdc4cbcd640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.191+0000 7fdc4cbcd640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.192+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 >> v1:172.21.15.153:6816/2737028074 conn(0x7fdc2806dbc0 legacy=0x7fdc28070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.192+0000 7fdc4cbcd640 10 monclient: shutdown 2024-03-29T11:35:04.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.192+0000 7fdc4cbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:04.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.192+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 >> v1:172.21.15.33:6790/0 conn(0x7fdc480a9d70 legacy=0x7fdc481dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.192+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 shutdown_connections 2024-03-29T11:35:04.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.192+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 >> 172.21.15.33:0/2074789602 conn(0x7fdc48066a00 msgr2=0x7fdc480b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:04.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.195+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 shutdown_connections 2024-03-29T11:35:04.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.195+0000 7fdc4cbcd640 1 -- 172.21.15.33:0/2074789602 wait complete. 2024-03-29T11:35:04.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:04.195+0000 7fdc4cbcd640 1 librados: shutdown 2024-03-29T11:35:04.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:11.034 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:11.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.179+0000 7f0f619ab640 10 monclient: get_monmap_and_config 2024-03-29T11:35:11.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.179+0000 7f0f619ab640 10 monclient: build_initial_monmap 2024-03-29T11:35:11.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.179+0000 7f0f619ab640 10 monclient: monmap: 2024-03-29T11:35:11.177 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:11.177 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:11.177 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:11.180232+0000 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:11.180232+0000 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.180+0000 7f0f619ab640 1 Processor -- start 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.180+0000 7f0f619ab640 1 -- start start 2024-03-29T11:35:11.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.180+0000 7f0f619ab640 10 monclient: init 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.181+0000 7f0f619ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.181+0000 7f0f619ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.181+0000 7f0f619ab640 10 monclient(hunting): picked mon.noname-a con 0x7f0f54006a70 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.181+0000 7f0f619ab640 10 monclient(hunting): picked mon.noname-c con 0x7f0f5402b100 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.182+0000 7f0f619ab640 10 monclient(hunting): picked mon.noname-b con 0x7f0f54029c30 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.182+0000 7f0f619ab640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5402a270 con 0x7f0f54006a70 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.182+0000 7f0f619ab640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5402c440 con 0x7f0f5402b100 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.182+0000 7f0f619ab640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f540c9440 con 0x7f0f54029c30 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.182+0000 7f0f619ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:11.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.182+0000 7f0f619ab640 10 monclient(hunting): authenticate will time out at 867.445917s 2024-03-29T11:35:11.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.184+0000 7f0f5bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0f5402b100 0x7f0f540cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:60654/0 (socket says 172.21.15.33:60654) 2024-03-29T11:35:11.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.184+0000 7f0f5bfff640 1 -- 172.21.15.33:0/3581219044 learned_addr learned my addr 172.21.15.33:0/3581219044 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:11.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.184+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287777117 0 0) 0x7f0f5402a270 con 0x7f0f54006a70 2024-03-29T11:35:11.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.184+0000 7f0f5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.184+0000 7f0f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.184+0000 7f0f5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.184+0000 7f0f5b7fe640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:35:11.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f4c003520 con 0x7f0f54006a70 2024-03-29T11:35:11.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430904379 0 0) 0x7f0f5402c440 con 0x7f0f5402b100 2024-03-29T11:35:11.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f5402a270 con 0x7f0f5402b100 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2283018562 0 0) 0x7f0f4c003520 con 0x7f0f54006a70 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 >> v1:172.21.15.33:6790/0 conn(0x7f0f54029c30 legacy=0x7f0f540d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 >> v1:172.21.15.153:6789/0 conn(0x7f0f5402b100 legacy=0x7f0f540cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f540c95d0 con 0x7f0f54006a70 2024-03-29T11:35:11.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:11.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:11.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:11.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0f500015c0 con 0x7f0f54006a70 2024-03-29T11:35:11.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:11.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: dump: 2024-03-29T11:35:11.188 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f619ab640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f619ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f50001720 con 0x7f0f54006a70 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f619ab640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:11.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.185+0000 7f0f619ab640 4 set_mon_vals no callback set 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f5b7fe640 1 -- 172.21.15.33:0/3581219044 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0f50001b30 con 0x7f0f54006a70 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f5b7fe640 10 monclient: dump: 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:11.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f619ab640 10 monclient: shutdown 2024-03-29T11:35:11.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f619ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:11.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f619ab640 1 -- 172.21.15.33:0/3581219044 >> v1:172.21.15.33:6789/0 conn(0x7f0f54006a70 legacy=0x7f0f540c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f619ab640 1 -- 172.21.15.33:0/3581219044 shutdown_connections 2024-03-29T11:35:11.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f619ab640 1 -- 172.21.15.33:0/3581219044 >> 172.21.15.33:0/3581219044 conn(0x7f0f540058f0 msgr2=0x7f0f54005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:11.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f619ab640 1 -- 172.21.15.33:0/3581219044 shutdown_connections 2024-03-29T11:35:11.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.186+0000 7f0f619ab640 1 -- 172.21.15.33:0/3581219044 wait complete. 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.187+0000 7f0f619ab640 10 monclient: build_initial_monmap 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.187+0000 7f0f619ab640 10 monclient: monmap: 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:11.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.188+0000 7f0f619ab640 1 librados: starting msgr at 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.188+0000 7f0f619ab640 1 librados: starting objecter 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.188+0000 7f0f619ab640 1 Processor -- start 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.190+0000 7f0f619ab640 1 -- start start 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.190+0000 7f0f619ab640 1 librados: setting wanted keys 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.190+0000 7f0f619ab640 1 librados: calling monclient init 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.190+0000 7f0f619ab640 10 monclient: init 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.191+0000 7f0f619ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.191+0000 7f0f619ab640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.191+0000 7f0f619ab640 10 monclient(hunting): picked mon.noname-a con 0x7f0f54029c30 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.191+0000 7f0f619ab640 10 monclient(hunting): picked mon.noname-c con 0x7f0f54031c90 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.191+0000 7f0f619ab640 10 monclient(hunting): picked mon.noname-b con 0x7f0f54161a50 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.191+0000 7f0f619ab640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f54158750 con 0x7f0f54029c30 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.191+0000 7f0f619ab640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f54157660 con 0x7f0f54161a50 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.192+0000 7f0f619ab640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f54032730 con 0x7f0f54031c90 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.192+0000 7f0f619ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:11.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.192+0000 7f0f619ab640 10 monclient(hunting): authenticate will time out at 867.456336s 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.192+0000 7f0f611aa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0f54161a50 0x7f0f5416bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:34406/0 (socket says 172.21.15.33:34406) 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.192+0000 7f0f611aa640 1 -- 172.21.15.33:0/2750819905 learned_addr learned my addr 172.21.15.33:0/2750819905 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.193+0000 7f0f597fa640 10 client.?.objecter ms_handle_connect 0x7f0f54161a50 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.193+0000 7f0f597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974381682 0 0) 0x7f0f54157660 con 0x7f0f54161a50 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f40004eb0 con 0x7f0f54161a50 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 10 client.?.objecter ms_handle_connect 0x7f0f54029c30 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.194+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545902822 0 0) 0x7f0f54158750 con 0x7f0f54029c30 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f54157660 con 0x7f0f54029c30 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3003522569 0 0) 0x7f0f40004eb0 con 0x7f0f54161a50 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f54158750 con 0x7f0f54161a50 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0f5c04fdd0 con 0x7f0f54161a50 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): dump: 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:11.200 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3650822560 0 0) 0x7f0f54157660 con 0x7f0f54029c30 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f40004eb0 con 0x7f0f54029c30 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0f50001a30 con 0x7f0f54029c30 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.195+0000 7f0f597fa640 10 monclient(hunting): dump: 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.201 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.196+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3465335075 0 0) 0x7f0f54158750 con 0x7f0f54161a50 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.196+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 >> v1:172.21.15.33:6790/0 conn(0x7f0f54031c90 legacy=0x7f0f54161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.196+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 >> v1:172.21.15.33:6789/0 conn(0x7f0f54029c30 legacy=0x7f0f54158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.196+0000 7f0f597fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.196+0000 7f0f597fa640 1 monclient: found mon.b 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.196+0000 7f0f597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f54158980 con 0x7f0f54161a50 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f597fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f597fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:11.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 10 monclient: _renew_subs 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f54161ea0 con 0x7f0f54161a50 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 10 monclient: _renew_subs 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5416d410 con 0x7f0f54161a50 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f619ab640 1 librados: init done 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.197+0000 7f0f597fa640 10 client.4240.objecter ms_handle_connect 0x7f0f54031c90 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.198+0000 7f0f597fa640 10 client.4240.objecter resend_mon_ops 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.198+0000 7f0f619ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.198+0000 7f0f619ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.198+0000 7f0f619ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.198+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f5c04f400 con 0x7f0f54161a50 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.198+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f54004cd0 con 0x7f0f54161a50 2024-03-29T11:35:11.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.199+0000 7f0f597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.199+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0f5c075710 con 0x7f0f54161a50 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.199+0000 7f0f597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.199+0000 7f0f597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.199+0000 7f0f597fa640 10 monclient: dump: 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:11.204 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.200+0000 7f0f5affd640 4 set_mon_vals no callback set 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.201+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f0f5c067bf0 con 0x7f0f54161a50 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.201+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f0f5c0ff5f0 con 0x7f0f54161a50 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.201+0000 7f0f597fa640 10 client.4240.objecter ms_dispatch 0x7f0f5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.201+0000 7f0f609a9640 10 monclient: get_auth_request con 0x7f0f4006dc90 auth_method 0 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.201+0000 7f0f597fa640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.202+0000 7f0f597fa640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.202+0000 7f0f597fa640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.203+0000 7f0f597fa640 10 client.4240.objecter ms_handle_connect 0x7f0f4006dc90 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.208+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f5c0caed0 con 0x7f0f54161a50 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.208+0000 7f0f597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.208+0000 7f0f597fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:11.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.532+0000 7f0f619ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:11.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.532+0000 7f0f619ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:11.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.532+0000 7f0f619ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:11.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.532+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f540086f0 con 0x7f0f54161a50 2024-03-29T11:35:11.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.534+0000 7f0f597fa640 1 -- 172.21.15.33:0/2750819905 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f0f5c0cce00 con 0x7f0f54161a50 2024-03-29T11:35:11.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.534+0000 7f0f597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:11.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.534+0000 7f0f597fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:11.532 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:11.532 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:11.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 10 librados: watch_flush enter 2024-03-29T11:35:11.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 10 librados: watch_flush exit 2024-03-29T11:35:11.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:35:11.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:35:11.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 >> v1:172.21.15.153:6816/2737028074 conn(0x7f0f4006dc90 legacy=0x7f0f40070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 10 monclient: shutdown 2024-03-29T11:35:11.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:11.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 >> v1:172.21.15.153:6789/0 conn(0x7f0f54161a50 legacy=0x7f0f5416bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 shutdown_connections 2024-03-29T11:35:11.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.538+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 >> 172.21.15.33:0/2750819905 conn(0x7f0f540058f0 msgr2=0x7f0f540c4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:11.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.541+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 shutdown_connections 2024-03-29T11:35:11.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.541+0000 7f0f619ab640 1 -- 172.21.15.33:0/2750819905 wait complete. 2024-03-29T11:35:11.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:11.541+0000 7f0f619ab640 1 librados: shutdown 2024-03-29T11:35:11.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:18.363 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:18.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.519+0000 7f57304b3640 10 monclient: get_monmap_and_config 2024-03-29T11:35:18.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.519+0000 7f57304b3640 10 monclient: build_initial_monmap 2024-03-29T11:35:18.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.519+0000 7f57304b3640 10 monclient: monmap: 2024-03-29T11:35:18.518 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:18.518 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:18.518 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:18.521235+0000 2024-03-29T11:35:18.518 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:18.521235+0000 2024-03-29T11:35:18.519 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:18.519 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:18.519 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:18.519 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:18.519 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:18.519 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:18.522 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.523+0000 7f57304b3640 1 Processor -- start 2024-03-29T11:35:18.522 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.523+0000 7f57304b3640 1 -- start start 2024-03-29T11:35:18.522 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.523+0000 7f57304b3640 10 monclient: init 2024-03-29T11:35:18.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.524+0000 7f57304b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:18.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.524+0000 7f57304b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:18.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 10 monclient(hunting): picked mon.noname-a con 0x7f57280b1f60 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 10 monclient(hunting): picked mon.noname-c con 0x7f5728071eb0 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5728143d10 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572806ef80 con 0x7f57280b1f60 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572806f260 con 0x7f5728071eb0 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57280b40c0 con 0x7f5728143d10 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.527+0000 7f57304b3640 10 monclient(hunting): authenticate will time out at 874.791934s 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.528+0000 7f572e24f640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f57280b1f60 0x7f5728143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49049/0 (socket says 172.21.15.33:49049) 2024-03-29T11:35:18.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.529+0000 7f572e24f640 1 -- 172.21.15.33:0/1657050224 learned_addr learned my addr 172.21.15.33:0/1657050224 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:18.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.530+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258841451 0 0) 0x7f572806ef80 con 0x7f57280b1f60 2024-03-29T11:35:18.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.530+0000 7f572d24d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:18.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.530+0000 7f572d24d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:18.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.530+0000 7f572d24d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:18.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.530+0000 7f572d24d640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:35:18.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.530+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5714003520 con 0x7f57280b1f60 2024-03-29T11:35:18.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482286686 0 0) 0x7f57280b40c0 con 0x7f5728143d10 2024-03-29T11:35:18.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:18.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:18.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:18.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:35:18.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572806ef80 con 0x7f5728143d10 2024-03-29T11:35:18.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 367935820 0 0) 0x7f572806ef80 con 0x7f5728143d10 2024-03-29T11:35:18.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 >> v1:172.21.15.153:6789/0 conn(0x7f5728071eb0 legacy=0x7f5728146470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:35:18.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 >> v1:172.21.15.33:6789/0 conn(0x7f57280b1f60 legacy=0x7f5728143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient: _finish_hunting 0 2024-03-29T11:35:18.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 monclient: found mon.noname-b 2024-03-29T11:35:18.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:18.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5728155d70 con 0x7f5728143d10 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient: _finish_auth 0 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient: _check_auth_tickets 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f57200019f0 con 0x7f5728143d10 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.531+0000 7f572d24d640 10 monclient: dump: 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.532 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f57304b3640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f57304b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5720001b50 con 0x7f5728143d10 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f572d24d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f572d24d640 1 -- 172.21.15.33:0/1657050224 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5720001fa0 con 0x7f5728143d10 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f572d24d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f572d24d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.532+0000 7f572d24d640 10 monclient: dump: 2024-03-29T11:35:18.533 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:18.534 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:18.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.533+0000 7f57304b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:18.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.533+0000 7f57304b3640 4 set_mon_vals no callback set 2024-03-29T11:35:18.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.534+0000 7f57304b3640 10 monclient: shutdown 2024-03-29T11:35:18.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.534+0000 7f57304b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:18.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.534+0000 7f57304b3640 1 -- 172.21.15.33:0/1657050224 >> v1:172.21.15.33:6790/0 conn(0x7f5728143d10 legacy=0x7f5728146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.534+0000 7f57304b3640 1 -- 172.21.15.33:0/1657050224 shutdown_connections 2024-03-29T11:35:18.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.534+0000 7f57304b3640 1 -- 172.21.15.33:0/1657050224 >> 172.21.15.33:0/1657050224 conn(0x7f5728066a00 msgr2=0x7f5728066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:18.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.534+0000 7f57304b3640 1 -- 172.21.15.33:0/1657050224 shutdown_connections 2024-03-29T11:35:18.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.535+0000 7f57304b3640 1 -- 172.21.15.33:0/1657050224 wait complete. 2024-03-29T11:35:18.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.536+0000 7f57304b3640 10 monclient: build_initial_monmap 2024-03-29T11:35:18.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.536+0000 7f57304b3640 10 monclient: monmap: 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.537+0000 7f57304b3640 1 librados: starting msgr at 2024-03-29T11:35:18.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.537+0000 7f57304b3640 1 librados: starting objecter 2024-03-29T11:35:18.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.537+0000 7f57304b3640 1 Processor -- start 2024-03-29T11:35:18.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.537+0000 7f57304b3640 1 -- start start 2024-03-29T11:35:18.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.537+0000 7f57304b3640 1 librados: setting wanted keys 2024-03-29T11:35:18.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.537+0000 7f57304b3640 1 librados: calling monclient init 2024-03-29T11:35:18.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.537+0000 7f57304b3640 10 monclient: init 2024-03-29T11:35:18.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:18.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:18.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 10 monclient(hunting): picked mon.noname-c con 0x7f57280b1f60 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:18.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5728143d10 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:18.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 10 monclient(hunting): picked mon.noname-a con 0x7f57281e4520 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:18.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5728149d60 con 0x7f57281e4520 2024-03-29T11:35:18.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5728140840 con 0x7f5728143d10 2024-03-29T11:35:18.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5728140ac0 con 0x7f57280b1f60 2024-03-29T11:35:18.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:18.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.538+0000 7f57304b3640 10 monclient(hunting): authenticate will time out at 874.803490s 2024-03-29T11:35:18.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.542+0000 7f572e24f640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f57280b1f60 0x7f57281e3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:59173/0 (socket says 172.21.15.33:59173) 2024-03-29T11:35:18.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.542+0000 7f572e24f640 1 -- 172.21.15.33:0/3480738938 learned_addr learned my addr 172.21.15.33:0/3480738938 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:18.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.542+0000 7f571effd640 10 client.?.objecter ms_handle_connect 0x7f57280b1f60 2024-03-29T11:35:18.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.542+0000 7f571effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:18.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.543+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493850593 0 0) 0x7f5728140ac0 con 0x7f57280b1f60 2024-03-29T11:35:18.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:18.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:18.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:18.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:35:18.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c004eb0 con 0x7f57280b1f60 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 client.?.objecter ms_handle_connect 0x7f5728143d10 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342815782 0 0) 0x7f5728140840 con 0x7f5728143d10 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.544+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728140ac0 con 0x7f5728143d10 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3186755311 0 0) 0x7f570c004eb0 con 0x7f57280b1f60 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5728140840 con 0x7f57280b1f60 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5724001a70 con 0x7f57280b1f60 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:18.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient(hunting): dump: 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2060940746 0 0) 0x7f5728140840 con 0x7f57280b1f60 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 >> v1:172.21.15.153:6789/0 conn(0x7f5728143d10 legacy=0x7f572813fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 >> v1:172.21.15.33:6789/0 conn(0x7f57281e4520 legacy=0x7f5728140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:18.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 monclient: found mon.c 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5728148d90 con 0x7f57280b1f60 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient: _finish_auth 0 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f571effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 10 monclient: _renew_subs 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572814a1f0 con 0x7f57280b1f60 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 10 monclient: _renew_subs 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5728070ae0 con 0x7f57280b1f60 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.545+0000 7f57304b3640 1 librados: init done 2024-03-29T11:35:18.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.546+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57240047e0 con 0x7f57280b1f60 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.546+0000 7f571effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.546+0000 7f572ca4c640 4 set_mon_vals no callback set 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.546+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5724004af0 con 0x7f57280b1f60 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.546+0000 7f571effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.546+0000 7f571effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.546+0000 7f571effd640 10 monclient: dump: 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:18.549 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.547+0000 7f57304b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.547+0000 7f57304b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.547+0000 7f57304b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.547+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5710005080 con 0x7f57280b1f60 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.548+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f572401aac0 con 0x7f57280b1f60 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.548+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f5724084b10 con 0x7f57280b1f60 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.548+0000 7f571effd640 10 client.4262.objecter ms_dispatch 0x7f5728000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.548+0000 7f571effd640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.548+0000 7f571effd640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.548+0000 7f571effd640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.549+0000 7f572da4e640 10 monclient: get_auth_request con 0x7f570c06dd30 auth_method 0 2024-03-29T11:35:18.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.551+0000 7f571effd640 10 client.4262.objecter ms_handle_connect 0x7f570c06dd30 2024-03-29T11:35:18.558 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.560+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5724050220 con 0x7f57280b1f60 2024-03-29T11:35:18.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.560+0000 7f571effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:18.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.560+0000 7f571effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:18.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.902+0000 7f57304b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.902+0000 7f57304b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.902+0000 7f57304b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:18.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.902+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5710005330 con 0x7f57280b1f60 2024-03-29T11:35:18.906 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:18.906 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:18.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.904+0000 7f571effd640 1 -- 172.21.15.33:0/3480738938 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f5724052320 con 0x7f57280b1f60 2024-03-29T11:35:18.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.904+0000 7f571effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.904+0000 7f571effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:18.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.909+0000 7f57304b3640 10 librados: watch_flush enter 2024-03-29T11:35:18.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.909+0000 7f57304b3640 10 librados: watch_flush exit 2024-03-29T11:35:18.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.909+0000 7f57304b3640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:35:18.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.909+0000 7f57304b3640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:35:18.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 >> v1:172.21.15.153:6816/2737028074 conn(0x7f570c06dd30 legacy=0x7f570c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 10 monclient: shutdown 2024-03-29T11:35:18.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:18.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 >> v1:172.21.15.33:6790/0 conn(0x7f57280b1f60 legacy=0x7f57281e3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 shutdown_connections 2024-03-29T11:35:18.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 >> 172.21.15.33:0/3480738938 conn(0x7f5728066a00 msgr2=0x7f57280b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:18.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 shutdown_connections 2024-03-29T11:35:18.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 1 -- 172.21.15.33:0/3480738938 wait complete. 2024-03-29T11:35:18.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:18.910+0000 7f57304b3640 1 librados: shutdown 2024-03-29T11:35:18.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:25.732 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:25.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.887+0000 7fa3b38c3640 10 monclient: get_monmap_and_config 2024-03-29T11:35:25.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.887+0000 7fa3b38c3640 10 monclient: build_initial_monmap 2024-03-29T11:35:25.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.887+0000 7fa3b38c3640 10 monclient: monmap: 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:25.889447+0000 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:25.889447+0000 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:25.888 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:25.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.889+0000 7fa3b38c3640 1 Processor -- start 2024-03-29T11:35:25.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.889+0000 7fa3b38c3640 1 -- start start 2024-03-29T11:35:25.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.889+0000 7fa3b38c3640 10 monclient: init 2024-03-29T11:35:25.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac0a9930 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac149dc0 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac14ff40 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0aae60 con 0x7fa3ac149dc0 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0b4310 con 0x7fa3ac14ff40 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0afdd0 con 0x7fa3ac0a9930 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:25.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.890+0000 7fa3b38c3640 10 monclient(hunting): authenticate will time out at 882.155397s 2024-03-29T11:35:25.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.896+0000 7fa3b0e5e640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fa3ac149dc0 0x7fa3ac14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:43473/0 (socket says 172.21.15.33:43473) 2024-03-29T11:35:25.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.896+0000 7fa3b0e5e640 1 -- 172.21.15.33:0/4117205743 learned_addr learned my addr 172.21.15.33:0/4117205743 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.898+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177977049 0 0) 0x7fa3ac0aae60 con 0x7fa3ac149dc0 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.898+0000 7fa3a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.898+0000 7fa3a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.898+0000 7fa3a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.898+0000 7fa3a3fff640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa398003520 con 0x7fa3ac149dc0 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318919278 0 0) 0x7fa3ac0afdd0 con 0x7fa3ac0a9930 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:35:25.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac0aae60 con 0x7fa3ac0a9930 2024-03-29T11:35:25.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119614547 0 0) 0x7fa3ac0b4310 con 0x7fa3ac14ff40 2024-03-29T11:35:25.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac0afdd0 con 0x7fa3ac14ff40 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.899+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4140351379 0 0) 0x7fa3ac0aae60 con 0x7fa3ac0a9930 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 >> v1:172.21.15.153:6789/0 conn(0x7fa3ac14ff40 legacy=0x7fa3ac152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 >> v1:172.21.15.33:6789/0 conn(0x7fa3ac149dc0 legacy=0x7fa3ac14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 1 monclient: found mon.noname-b 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac14a950 con 0x7fa3ac0a9930 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3b38c3640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3b38c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:25.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa3a80015f0 con 0x7fa3ac0a9930 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: dump: 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3b38c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:25.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3a8001750 con 0x7fa3ac0a9930 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3b38c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3b38c3640 4 set_mon_vals no callback set 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 1 -- 172.21.15.33:0/4117205743 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa3a8001ba0 con 0x7fa3ac0a9930 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.900+0000 7fa3a3fff640 10 monclient: dump: 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:25.902 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:25.903 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:25.903 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:25.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.901+0000 7fa3b38c3640 10 monclient: shutdown 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.901+0000 7fa3b38c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.901+0000 7fa3b38c3640 1 -- 172.21.15.33:0/4117205743 >> v1:172.21.15.33:6790/0 conn(0x7fa3ac0a9930 legacy=0x7fa3ac143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.901+0000 7fa3b38c3640 1 -- 172.21.15.33:0/4117205743 shutdown_connections 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.901+0000 7fa3b38c3640 1 -- 172.21.15.33:0/4117205743 >> 172.21.15.33:0/4117205743 conn(0x7fa3ac0b35e0 msgr2=0x7fa3ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.902+0000 7fa3b38c3640 1 -- 172.21.15.33:0/4117205743 shutdown_connections 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.902+0000 7fa3b38c3640 1 -- 172.21.15.33:0/4117205743 wait complete. 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.903+0000 7fa3b38c3640 10 monclient: build_initial_monmap 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.903+0000 7fa3b38c3640 10 monclient: monmap: 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:25.904 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 1 librados: starting msgr at 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 1 librados: starting objecter 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 1 Processor -- start 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 1 -- start start 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 1 librados: setting wanted keys 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 1 librados: calling monclient init 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 10 monclient: init 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.904+0000 7fa3b38c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:25.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac149dc0 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac14ff40 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac148090 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac149290 con 0x7fa3ac148090 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac147de0 con 0x7fa3ac14ff40 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0ab570 con 0x7fa3ac149dc0 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:25.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.905+0000 7fa3b38c3640 10 monclient(hunting): authenticate will time out at 882.169757s 2024-03-29T11:35:25.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.908+0000 7fa3b165f640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fa3ac149dc0 0x7fa3ac13a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:50091/0 (socket says 172.21.15.33:50091) 2024-03-29T11:35:25.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.908+0000 7fa3b165f640 1 -- 172.21.15.33:0/3640388434 learned_addr learned my addr 172.21.15.33:0/3640388434 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.910+0000 7fa3a1ffb640 10 client.?.objecter ms_handle_connect 0x7fa3ac149dc0 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.910+0000 7fa3a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.910+0000 7fa3a1ffb640 10 client.?.objecter ms_handle_connect 0x7fa3ac14ff40 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.910+0000 7fa3a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.910+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598586061 0 0) 0x7fa3ac0ab570 con 0x7fa3ac149dc0 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa390004eb0 con 0x7fa3ac149dc0 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 client.?.objecter ms_handle_connect 0x7fa3ac148090 2024-03-29T11:35:25.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1105064080 0 0) 0x7fa390004eb0 con 0x7fa3ac149dc0 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3ac0ab570 con 0x7fa3ac149dc0 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa3a8001ba0 con 0x7fa3ac149dc0 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.913 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.911+0000 7fa3a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978625105 0 0) 0x7fa3ac147de0 con 0x7fa3ac14ff40 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa390004eb0 con 0x7fa3ac14ff40 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307326001 0 0) 0x7fa3ac149290 con 0x7fa3ac148090 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac147de0 con 0x7fa3ac148090 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 60935580 0 0) 0x7fa3ac147de0 con 0x7fa3ac148090 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3ac149290 con 0x7fa3ac148090 2024-03-29T11:35:25.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa3a40019c0 con 0x7fa3ac148090 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.912+0000 7fa3a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:25.916 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2279659488 0 0) 0x7fa3ac149290 con 0x7fa3ac148090 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 >> v1:172.21.15.33:6790/0 conn(0x7fa3ac149dc0 legacy=0x7fa3ac13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 >> v1:172.21.15.153:6789/0 conn(0x7fa3ac14ff40 legacy=0x7fa3ac0aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 1 monclient: found mon.a 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0ab1d0 con 0x7fa3ac148090 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:25.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:25.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3a4001b20 con 0x7fa3ac148090 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa3a4003970 con 0x7fa3ac148090 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3a1ffb640 10 monclient: dump: 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:25.920 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.913+0000 7fa3b38c3640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 10 monclient: _renew_subs 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac1487e0 con 0x7fa3ac148090 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 10 monclient: _renew_subs 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac13aa40 con 0x7fa3ac148090 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.914+0000 7fa3b38c3640 1 librados: init done 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.915+0000 7fa3a2ffd640 4 set_mon_vals no callback set 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.917+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fa3a4019940 con 0x7fa3ac148090 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3b38c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3b38c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3b38c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c005080 con 0x7fa3ac148090 2024-03-29T11:35:25.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fa3a4083900 con 0x7fa3ac148090 2024-03-29T11:35:25.922 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3a1ffb640 10 client.4269.objecter ms_dispatch 0x7fa3ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:25.922 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3a1ffb640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:25.922 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.918+0000 7fa3a1ffb640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:25.922 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.919+0000 7fa3a1ffb640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T11:35:25.922 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.919+0000 7fa3b165f640 10 monclient: get_auth_request con 0x7fa39006df90 auth_method 0 2024-03-29T11:35:25.922 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.921+0000 7fa3a1ffb640 10 client.4269.objecter ms_handle_connect 0x7fa39006df90 2024-03-29T11:35:25.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.931+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3a404f1e0 con 0x7fa3ac148090 2024-03-29T11:35:25.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.931+0000 7fa3a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:25.931+0000 7fa3a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:26.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.254+0000 7fa3b38c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:26.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.254+0000 7fa3b38c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:26.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.254+0000 7fa3b38c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:35:26.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.254+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa38c005330 con 0x7fa3ac148090 2024-03-29T11:35:26.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.255+0000 7fa3a1ffb640 1 -- 172.21.15.33:0/3640388434 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fa3a4051110 con 0x7fa3ac148090 2024-03-29T11:35:26.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.255+0000 7fa3a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:26.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.255+0000 7fa3a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:26.254 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:26.254 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:26.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.258+0000 7fa3b38c3640 10 librados: watch_flush enter 2024-03-29T11:35:26.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 10 librados: watch_flush exit 2024-03-29T11:35:26.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T11:35:26.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 10 client.4269.objecter successfully canceled tick 2024-03-29T11:35:26.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 >> v1:172.21.15.153:6816/2737028074 conn(0x7fa39006df90 legacy=0x7fa390070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 10 monclient: shutdown 2024-03-29T11:35:26.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:26.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 >> v1:172.21.15.33:6789/0 conn(0x7fa3ac148090 legacy=0x7fa3ac0ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:26.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 shutdown_connections 2024-03-29T11:35:26.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 >> 172.21.15.33:0/3640388434 conn(0x7fa3ac0b35e0 msgr2=0x7fa3ac14a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:26.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.260+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 shutdown_connections 2024-03-29T11:35:26.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.261+0000 7fa3b38c3640 1 -- 172.21.15.33:0/3640388434 wait complete. 2024-03-29T11:35:26.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:26.261+0000 7fa3b38c3640 1 librados: shutdown 2024-03-29T11:35:26.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:33.084 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:33.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.237+0000 7f0353fff640 10 monclient: get_monmap_and_config 2024-03-29T11:35:33.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.237+0000 7f0353fff640 10 monclient: build_initial_monmap 2024-03-29T11:35:33.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.237+0000 7f0353fff640 10 monclient: monmap: 2024-03-29T11:35:33.236 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:33.236 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:33.238763+0000 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:33.238763+0000 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:33.237 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.238+0000 7f0353fff640 1 Processor -- start 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.238+0000 7f0353fff640 1 -- start start 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.238+0000 7f0353fff640 10 monclient: init 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.239+0000 7f0353fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.239+0000 7f0353fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.239+0000 7f0353fff640 10 monclient(hunting): picked mon.noname-a con 0x7f034c006e30 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.239+0000 7f0353fff640 10 monclient(hunting): picked mon.noname-c con 0x7f034c035e00 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.241+0000 7f0353fff640 10 monclient(hunting): picked mon.noname-b con 0x7f034c0c9670 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.241+0000 7f0353fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c035970 con 0x7f034c006e30 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.241+0000 7f0353fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c036280 con 0x7f034c035e00 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.241+0000 7f0353fff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c035640 con 0x7f034c0c9670 2024-03-29T11:35:33.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.241+0000 7f0353fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:33.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.241+0000 7f0353fff640 10 monclient(hunting): authenticate will time out at 889.506323s 2024-03-29T11:35:33.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f03537fe640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f034c0c9670 0x7f034c0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:52055/0 (socket says 172.21.15.33:52055) 2024-03-29T11:35:33.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f03537fe640 1 -- 172.21.15.33:0/2183793213 learned_addr learned my addr 172.21.15.33:0/2183793213 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:33.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698028341 0 0) 0x7f034c035640 con 0x7f034c0c9670 2024-03-29T11:35:33.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:33.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033c003520 con 0x7f034c0c9670 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3270347178 0 0) 0x7f033c003520 con 0x7f034c0c9670 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 >> v1:172.21.15.153:6789/0 conn(0x7f034c035e00 legacy=0x7f034c0c8f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.248+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 >> v1:172.21.15.33:6789/0 conn(0x7f034c006e30 legacy=0x7f034c0cbcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 1 monclient: found mon.noname-b 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c034e20 con 0x7f034c0c9670 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:33.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f035404f730 con 0x7f034c0c9670 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: dump: 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:33.253 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0354075f10 con 0x7f034c0c9670 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 1 -- 172.21.15.33:0/2183793213 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f03540758b0 con 0x7f034c0c9670 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.249+0000 7f0351ffb640 10 monclient: dump: 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.254 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.251+0000 7f0353fff640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.251+0000 7f0353fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.251+0000 7f0353fff640 4 set_mon_vals no callback set 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.252+0000 7f0353fff640 10 monclient: shutdown 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.252+0000 7f0353fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.252+0000 7f0353fff640 1 -- 172.21.15.33:0/2183793213 >> v1:172.21.15.33:6790/0 conn(0x7f034c0c9670 legacy=0x7f034c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.252+0000 7f0353fff640 1 -- 172.21.15.33:0/2183793213 shutdown_connections 2024-03-29T11:35:33.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.252+0000 7f0353fff640 1 -- 172.21.15.33:0/2183793213 >> 172.21.15.33:0/2183793213 conn(0x7f034c005c00 msgr2=0x7f034c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:33.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.252+0000 7f0353fff640 1 -- 172.21.15.33:0/2183793213 shutdown_connections 2024-03-29T11:35:33.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.252+0000 7f0353fff640 1 -- 172.21.15.33:0/2183793213 wait complete. 2024-03-29T11:35:33.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.253+0000 7f0353fff640 10 monclient: build_initial_monmap 2024-03-29T11:35:33.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.253+0000 7f0353fff640 10 monclient: monmap: 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.253+0000 7f0353fff640 1 librados: starting msgr at 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.253+0000 7f0353fff640 1 librados: starting objecter 2024-03-29T11:35:33.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.254+0000 7f0353fff640 1 Processor -- start 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.254+0000 7f0353fff640 1 -- start start 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.254+0000 7f0353fff640 1 librados: setting wanted keys 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.254+0000 7f0353fff640 1 librados: calling monclient init 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.254+0000 7f0353fff640 10 monclient: init 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.255+0000 7f0353fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.255+0000 7f0353fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.255+0000 7f0353fff640 10 monclient(hunting): picked mon.noname-a con 0x7f034c006e30 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.255+0000 7f0353fff640 10 monclient(hunting): picked mon.noname-c con 0x7f034c035e00 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:33.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.257+0000 7f0353fff640 10 monclient(hunting): picked mon.noname-b con 0x7f034c0c9670 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.257+0000 7f0353fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c160a40 con 0x7f034c006e30 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.257+0000 7f0353fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c02fd70 con 0x7f034c0c9670 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.257+0000 7f0353fff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c031f10 con 0x7f034c035e00 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.257+0000 7f0353fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.257+0000 7f0353fff640 10 monclient(hunting): authenticate will time out at 889.522390s 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.261+0000 7f03537fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f034c0c9670 0x7f034c16b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:49973/0 (socket says 172.21.15.33:49973) 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.261+0000 7f03537fe640 1 -- 172.21.15.33:0/1226541984 learned_addr learned my addr 172.21.15.33:0/1226541984 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 client.?.objecter ms_handle_connect 0x7f034c0c9670 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341182060 0 0) 0x7f034c02fd70 con 0x7f034c0c9670 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:33.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0340004eb0 con 0x7f034c0c9670 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 client.?.objecter ms_handle_connect 0x7f034c035e00 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906767806 0 0) 0x7f034c031f10 con 0x7f034c035e00 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.265+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f034c02fd70 con 0x7f034c035e00 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.266+0000 7f033b7fe640 10 client.?.objecter ms_handle_connect 0x7f034c006e30 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.266+0000 7f033b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:33.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.266+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3522954374 0 0) 0x7f0340004eb0 con 0x7f034c0c9670 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.266+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f034c031f10 con 0x7f034c0c9670 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.266+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f035404f400 con 0x7f034c0c9670 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.266+0000 7f033b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient(hunting): dump: 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:33.269 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:33.270 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:33.270 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:33.270 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:33.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610879411 0 0) 0x7f034c160a40 con 0x7f034c006e30 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0340004eb0 con 0x7f034c006e30 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3167163081 0 0) 0x7f034c02fd70 con 0x7f034c035e00 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f034c160a40 con 0x7f034c035e00 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2621288734 0 0) 0x7f034c031f10 con 0x7f034c0c9670 2024-03-29T11:35:33.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 >> v1:172.21.15.33:6790/0 conn(0x7f034c035e00 legacy=0x7f034c164230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 >> v1:172.21.15.33:6789/0 conn(0x7f034c006e30 legacy=0x7f034c160ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 monclient: found mon.b 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c02ffe0 con 0x7f034c0c9670 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 10 monclient: _renew_subs 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:33.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c161640 con 0x7f034c0c9670 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 10 monclient: _renew_subs 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c164b20 con 0x7f034c0c9670 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f035404f730 con 0x7f034c0c9670 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f0353fff640 1 librados: init done 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.267+0000 7f033b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.268+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0354067d90 con 0x7f034c0c9670 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.268+0000 7f033b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.268+0000 7f033b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.268+0000 7f033b7fe640 10 monclient: dump: 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:33.274 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.269+0000 7f03517fa640 4 set_mon_vals no callback set 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.270+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f0354067ef0 con 0x7f034c0c9670 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.271+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f03540ff3e0 con 0x7f034c0c9670 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.271+0000 7f033b7fe640 10 client.4252.objecter ms_dispatch 0x7f0354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:33.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.271+0000 7f033b7fe640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.271+0000 7f033b7fe640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.271+0000 7f033b7fe640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.272+0000 7f0353fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.272+0000 7f0353fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.272+0000 7f0353fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.272+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0334005080 con 0x7f034c0c9670 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.272+0000 7f0352ffd640 10 monclient: get_auth_request con 0x7f034006df20 auth_method 0 2024-03-29T11:35:33.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.273+0000 7f033b7fe640 10 client.4252.objecter ms_handle_connect 0x7f034006df20 2024-03-29T11:35:33.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.282+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03540cacc0 con 0x7f034c0c9670 2024-03-29T11:35:33.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.282+0000 7f033b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:33.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.282+0000 7f033b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:33.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.608+0000 7f0353fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.608+0000 7f0353fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.608+0000 7f0353fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:33.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.608+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0334005330 con 0x7f034c0c9670 2024-03-29T11:35:33.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.610+0000 7f033b7fe640 1 -- 172.21.15.33:0/1226541984 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f0354068220 con 0x7f034c0c9670 2024-03-29T11:35:33.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.610+0000 7f033b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.610+0000 7f033b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:33.609 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:33.609 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:33.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.613+0000 7f0353fff640 10 librados: watch_flush enter 2024-03-29T11:35:33.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.615+0000 7f0353fff640 10 librados: watch_flush exit 2024-03-29T11:35:33.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.615+0000 7f0353fff640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T11:35:33.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.615+0000 7f0353fff640 10 client.4252.objecter successfully canceled tick 2024-03-29T11:35:33.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.615+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 >> v1:172.21.15.153:6816/2737028074 conn(0x7f034006df20 legacy=0x7f03400703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.615+0000 7f0353fff640 10 monclient: shutdown 2024-03-29T11:35:33.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.615+0000 7f0353fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:33.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.615+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 >> v1:172.21.15.153:6789/0 conn(0x7f034c0c9670 legacy=0x7f034c16b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.616+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 shutdown_connections 2024-03-29T11:35:33.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.616+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 >> 172.21.15.33:0/1226541984 conn(0x7f034c005c00 msgr2=0x7f034c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:33.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.618+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 shutdown_connections 2024-03-29T11:35:33.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.618+0000 7f0353fff640 1 -- 172.21.15.33:0/1226541984 wait complete. 2024-03-29T11:35:33.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:33.618+0000 7f0353fff640 1 librados: shutdown 2024-03-29T11:35:33.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:40.442 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.600+0000 7f61a49dc640 10 monclient: get_monmap_and_config 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.600+0000 7f61a49dc640 10 monclient: build_initial_monmap 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.600+0000 7f61a49dc640 10 monclient: monmap: 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:40.600747+0000 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:40.600747+0000 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:40.598 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:40.599 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:40.599 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:40.599 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:40.599 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:40.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.601+0000 7f61a49dc640 1 Processor -- start 2024-03-29T11:35:40.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.601+0000 7f61a49dc640 1 -- start start 2024-03-29T11:35:40.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.601+0000 7f61a49dc640 10 monclient: init 2024-03-29T11:35:40.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:40.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:40.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 10 monclient(hunting): picked mon.noname-a con 0x7f61980389e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:40.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 10 monclient(hunting): picked mon.noname-b con 0x7f6198038dc0 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:40.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6198030660 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:40.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61980c54c0 con 0x7f61980389e0 2024-03-29T11:35:40.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6198036430 con 0x7f6198030660 2024-03-29T11:35:40.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6198039840 con 0x7f6198038dc0 2024-03-29T11:35:40.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:40.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.602+0000 7f61a49dc640 10 monclient(hunting): authenticate will time out at 896.866113s 2024-03-29T11:35:40.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.608+0000 7f619f7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f61980389e0 0x7f6198033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:48297/0 (socket says 172.21.15.33:48297) 2024-03-29T11:35:40.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.608+0000 7f619f7fe640 1 -- 172.21.15.33:0/3113533487 learned_addr learned my addr 172.21.15.33:0/3113533487 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:40.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.608+0000 7f619effd640 1 --1- 172.21.15.33:0/3113533487 >> v1:172.21.15.33:6790/0 conn(0x7f6198038dc0 0x7f61980cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:59523/0 (socket says 172.21.15.33:59523) 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32509820 0 0) 0x7f61980c54c0 con 0x7f61980389e0 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6190003520 con 0x7f61980389e0 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239167832 0 0) 0x7f6198039840 con 0x7f6198038dc0 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.609+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61980c54c0 con 0x7f6198038dc0 2024-03-29T11:35:40.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861517669 0 0) 0x7f6190003520 con 0x7f61980389e0 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 >> v1:172.21.15.33:6790/0 conn(0x7f6198038dc0 legacy=0x7f61980cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 >> v1:172.21.15.153:6789/0 conn(0x7f6198030660 legacy=0x7f61980d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198035690 con 0x7f61980389e0 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f61a49dc640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f61a49dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f618c001690 con 0x7f61980389e0 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.610+0000 7f619e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.611+0000 7f619e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.611+0000 7f619e7fc640 10 monclient: dump: 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.611+0000 7f61a49dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.611+0000 7f619e7fc640 1 -- 172.21.15.33:0/3113533487 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f618c0018b0 con 0x7f61980389e0 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.611+0000 7f619e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.611+0000 7f61a49dc640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:40.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.611+0000 7f61a49dc640 4 set_mon_vals no callback set 2024-03-29T11:35:40.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.612+0000 7f61a49dc640 10 monclient: shutdown 2024-03-29T11:35:40.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.612+0000 7f61a49dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:40.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.612+0000 7f61a49dc640 1 -- 172.21.15.33:0/3113533487 >> v1:172.21.15.33:6789/0 conn(0x7f61980389e0 legacy=0x7f6198033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.612+0000 7f61a49dc640 1 -- 172.21.15.33:0/3113533487 shutdown_connections 2024-03-29T11:35:40.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.612+0000 7f61a49dc640 1 -- 172.21.15.33:0/3113533487 >> 172.21.15.33:0/3113533487 conn(0x7f6198005c00 msgr2=0x7f6198006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:40.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.615+0000 7f61a49dc640 1 -- 172.21.15.33:0/3113533487 shutdown_connections 2024-03-29T11:35:40.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.615+0000 7f61a49dc640 1 -- 172.21.15.33:0/3113533487 wait complete. 2024-03-29T11:35:40.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.618+0000 7f61a49dc640 10 monclient: build_initial_monmap 2024-03-29T11:35:40.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.618+0000 7f61a49dc640 10 monclient: monmap: 2024-03-29T11:35:40.616 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:40.616 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.618+0000 7f61a49dc640 1 librados: starting msgr at 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.618+0000 7f61a49dc640 1 librados: starting objecter 2024-03-29T11:35:40.617 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.618+0000 7f61a49dc640 1 Processor -- start 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.619+0000 7f61a49dc640 1 -- start start 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.619+0000 7f61a49dc640 1 librados: setting wanted keys 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.619+0000 7f61a49dc640 1 librados: calling monclient init 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.619+0000 7f61a49dc640 10 monclient: init 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6198030660 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 10 monclient(hunting): picked mon.noname-b con 0x7f619802ee40 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 10 monclient(hunting): picked mon.noname-a con 0x7f619815cc20 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6198160120 con 0x7f619815cc20 2024-03-29T11:35:40.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61981608d0 con 0x7f619802ee40 2024-03-29T11:35:40.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61981603b0 con 0x7f6198030660 2024-03-29T11:35:40.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:40.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.620+0000 7f61a49dc640 10 monclient(hunting): authenticate will time out at 896.883843s 2024-03-29T11:35:40.621 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.623+0000 7f619f7fe640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f6198030660 0x7f619815fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:59639/0 (socket says 172.21.15.33:59639) 2024-03-29T11:35:40.621 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.623+0000 7f619f7fe640 1 -- 172.21.15.33:0/211419026 learned_addr learned my addr 172.21.15.33:0/211419026 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:40.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.626+0000 7f618bfff640 10 client.?.objecter ms_handle_connect 0x7f6198030660 2024-03-29T11:35:40.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.626+0000 7f618bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.627+0000 7f618bfff640 10 client.?.objecter ms_handle_connect 0x7f619802ee40 2024-03-29T11:35:40.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.627+0000 7f618bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.627+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559698537 0 0) 0x7f61981603b0 con 0x7f6198030660 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.627+0000 7f618bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6184004eb0 con 0x7f6198030660 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359371308 0 0) 0x7f61981608d0 con 0x7f619802ee40 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:35:40.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61981603b0 con 0x7f619802ee40 2024-03-29T11:35:40.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250610345 0 0) 0x7f6184004eb0 con 0x7f6198030660 2024-03-29T11:35:40.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61981608d0 con 0x7f6198030660 2024-03-29T11:35:40.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f618c004980 con 0x7f6198030660 2024-03-29T11:35:40.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:40.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): dump: 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 926318644 0 0) 0x7f61981603b0 con 0x7f619802ee40 2024-03-29T11:35:40.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6184004eb0 con 0x7f619802ee40 2024-03-29T11:35:40.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.628+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f6194001850 con 0x7f619802ee40 2024-03-29T11:35:40.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.629+0000 7f618bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 10 monclient(hunting): dump: 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:40.630 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 10 client.?.objecter ms_handle_connect 0x7f619815cc20 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1248430032 0 0) 0x7f61981608d0 con 0x7f6198030660 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 >> v1:172.21.15.153:6789/0 conn(0x7f619802ee40 legacy=0x7f619815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 >> v1:172.21.15.33:6789/0 conn(0x7f619815cc20 legacy=0x7f6198166fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 1 monclient: found mon.c 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.630+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61980cad00 con 0x7f6198030660 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f618bfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f618bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:40.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f618bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:40.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:35:40.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 10 monclient: _renew_subs 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61981686e0 con 0x7f6198030660 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 10 monclient: _renew_subs 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6198168a50 con 0x7f6198030660 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.631+0000 7f61a49dc640 1 librados: init done 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.632+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f618c001aa0 con 0x7f6198030660 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.632+0000 7f618bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.632+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f618c001910 con 0x7f6198030660 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.632+0000 7f618bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.632+0000 7f618bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.632+0000 7f618bfff640 10 monclient: dump: 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:40.633 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.632+0000 7f619dffb640 4 set_mon_vals no callback set 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.634+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f618c01ada0 con 0x7f6198030660 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.634+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f618c0849e0 con 0x7f6198030660 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.634+0000 7f618bfff640 10 client.4280.objecter ms_dispatch 0x7f61a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.634+0000 7f618bfff640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.634+0000 7f618bfff640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:40.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.634+0000 7f618bfff640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T11:35:40.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.634+0000 7f619effd640 10 monclient: get_auth_request con 0x7f618406df00 auth_method 0 2024-03-29T11:35:40.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.635+0000 7f61a49dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.635+0000 7f61a49dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.635+0000 7f61a49dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:40.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.635+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6180005080 con 0x7f6198030660 2024-03-29T11:35:40.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.636+0000 7f618bfff640 10 client.4280.objecter ms_handle_connect 0x7f618406df00 2024-03-29T11:35:40.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.645+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f618c0502c0 con 0x7f6198030660 2024-03-29T11:35:40.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.645+0000 7f618bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.645+0000 7f618bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:40.978 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.980+0000 7f61a49dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.978 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.980+0000 7f61a49dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.978 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.980+0000 7f61a49dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:35:40.978 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.980+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6180005330 con 0x7f6198030660 2024-03-29T11:35:40.979 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.981+0000 7f618bfff640 1 -- 172.21.15.33:0/211419026 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f618c0521f0 con 0x7f6198030660 2024-03-29T11:35:40.979 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.981+0000 7f618bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.979 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.981+0000 7f618bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:40.980 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:40.980 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:40.983 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.985+0000 7f61a49dc640 10 librados: watch_flush enter 2024-03-29T11:35:40.984 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.986+0000 7f61a49dc640 10 librados: watch_flush exit 2024-03-29T11:35:40.984 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.986+0000 7f61a49dc640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T11:35:40.984 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.986+0000 7f61a49dc640 10 client.4280.objecter successfully canceled tick 2024-03-29T11:35:40.985 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.987+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 >> v1:172.21.15.153:6816/2737028074 conn(0x7f618406df00 legacy=0x7f61840703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.985 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.987+0000 7f61a49dc640 10 monclient: shutdown 2024-03-29T11:35:40.985 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.987+0000 7f61a49dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:40.985 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.987+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 >> v1:172.21.15.33:6790/0 conn(0x7f6198030660 legacy=0x7f619815fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.988+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 shutdown_connections 2024-03-29T11:35:40.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.988+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 >> 172.21.15.33:0/211419026 conn(0x7f6198005c00 msgr2=0x7f61980c4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:40.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.989+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 shutdown_connections 2024-03-29T11:35:40.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.989+0000 7f61a49dc640 1 -- 172.21.15.33:0/211419026 wait complete. 2024-03-29T11:35:40.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:40.989+0000 7f61a49dc640 1 librados: shutdown 2024-03-29T11:35:41.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:47.813 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:47.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.988+0000 7efeb7a06640 10 monclient: get_monmap_and_config 2024-03-29T11:35:47.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.988+0000 7efeb7a06640 10 monclient: build_initial_monmap 2024-03-29T11:35:47.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.988+0000 7efeb7a06640 10 monclient: monmap: 2024-03-29T11:35:47.986 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:47.986 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:47.986 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:47.988791+0000 2024-03-29T11:35:47.986 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:47.988791+0000 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.989+0000 7efeb7a06640 1 Processor -- start 2024-03-29T11:35:47.987 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.989+0000 7efeb7a06640 1 -- start start 2024-03-29T11:35:47.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.989+0000 7efeb7a06640 10 monclient: init 2024-03-29T11:35:47.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:47.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:47.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 10 monclient(hunting): picked mon.noname-a con 0x7efeb0071820 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:47.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 10 monclient(hunting): picked mon.noname-c con 0x7efeb00af660 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:47.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 10 monclient(hunting): picked mon.noname-b con 0x7efeb006ab70 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:47.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb00729c0 con 0x7efeb0071820 2024-03-29T11:35:47.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb00afaf0 con 0x7efeb00af660 2024-03-29T11:35:47.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb00b1110 con 0x7efeb006ab70 2024-03-29T11:35:47.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:47.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb7a06640 10 monclient(hunting): authenticate will time out at 904.254165s 2024-03-29T11:35:47.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb4fa1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7efeb00af660 0x7efeb0072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:43196/0 (socket says 172.21.15.33:43196) 2024-03-29T11:35:47.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.990+0000 7efeb4fa1640 1 -- 172.21.15.33:0/4247441174 learned_addr learned my addr 172.21.15.33:0/4247441174 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111630743 0 0) 0x7efeb00afaf0 con 0x7efeb00af660 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c003520 con 0x7efeb00af660 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447956374 0 0) 0x7efe9c003520 con 0x7efeb00af660 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 >> v1:172.21.15.33:6790/0 conn(0x7efeb006ab70 legacy=0x7efeb014d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 >> v1:172.21.15.33:6789/0 conn(0x7efeb0071820 legacy=0x7efeb0142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 monclient: found mon.noname-c 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00b12a0 con 0x7efeb00af660 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7efeac001690 con 0x7efeb00af660 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efea7fff640 10 monclient: dump: 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:47.991 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efeb7a06640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.991+0000 7efeb7a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeac0017f0 con 0x7efeb00af660 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efea7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efea7fff640 1 -- 172.21.15.33:0/4247441174 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7efeac001c00 con 0x7efeb00af660 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efea7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efea7fff640 10 monclient: dump: 2024-03-29T11:35:47.992 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:47.993 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:47.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efeb7a06640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:47.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efeb7a06640 4 set_mon_vals no callback set 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efeb7a06640 10 monclient: shutdown 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efeb7a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.992+0000 7efeb7a06640 1 -- 172.21.15.33:0/4247441174 >> v1:172.21.15.153:6789/0 conn(0x7efeb00af660 legacy=0x7efeb0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.993+0000 7efeb7a06640 1 -- 172.21.15.33:0/4247441174 shutdown_connections 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.993+0000 7efeb7a06640 1 -- 172.21.15.33:0/4247441174 >> 172.21.15.33:0/4247441174 conn(0x7efeb00b4930 msgr2=0x7efeb00b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.993+0000 7efeb7a06640 1 -- 172.21.15.33:0/4247441174 shutdown_connections 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.993+0000 7efeb7a06640 1 -- 172.21.15.33:0/4247441174 wait complete. 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.994+0000 7efeb7a06640 10 monclient: build_initial_monmap 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.994+0000 7efeb7a06640 10 monclient: monmap: 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:47.998 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.994+0000 7efeb7a06640 1 librados: starting msgr at 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.994+0000 7efeb7a06640 1 librados: starting objecter 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.994+0000 7efeb7a06640 1 Processor -- start 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.995+0000 7efeb7a06640 1 -- start start 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.995+0000 7efeb7a06640 1 librados: setting wanted keys 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.995+0000 7efeb7a06640 1 librados: calling monclient init 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.995+0000 7efeb7a06640 10 monclient: init 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:47.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 10 monclient(hunting): picked mon.noname-b con 0x7efeb006ab70 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 10 monclient(hunting): picked mon.noname-c con 0x7efeb0071820 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 10 monclient(hunting): picked mon.noname-a con 0x7efeb00af660 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb013e490 con 0x7efeb00af660 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb013ef10 con 0x7efeb006ab70 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb013f180 con 0x7efeb0071820 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb7a06640 10 monclient(hunting): authenticate will time out at 904.259885s 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb5fa3640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7efeb00af660 0x7efeb013ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:54482/0 (socket says 172.21.15.33:54482) 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efeb5fa3640 1 -- 172.21.15.33:0/3163375794 learned_addr learned my addr 172.21.15.33:0/3163375794 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efea5ffb640 10 client.?.objecter ms_handle_connect 0x7efeb00af660 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efea5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efea5ffb640 10 client.?.objecter ms_handle_connect 0x7efeb006ab70 2024-03-29T11:35:48.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.996+0000 7efea5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 client.?.objecter ms_handle_connect 0x7efeb0071820 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296262234 0 0) 0x7efeb013ef10 con 0x7efeb006ab70 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe94004eb0 con 0x7efeb006ab70 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642663909 0 0) 0x7efeb013e490 con 0x7efeb00af660 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb013ef10 con 0x7efeb00af660 2024-03-29T11:35:48.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1372723920 0 0) 0x7efe94004eb0 con 0x7efeb006ab70 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb013e490 con 0x7efeb006ab70 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7efea0001930 con 0x7efeb006ab70 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:48.002 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.997+0000 7efea5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044146639 0 0) 0x7efeb013f180 con 0x7efeb0071820 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe94004eb0 con 0x7efeb0071820 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736495051 0 0) 0x7efeb013ef10 con 0x7efeb00af660 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb013f180 con 0x7efeb00af660 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7efea8001be0 con 0x7efeb00af660 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:48.003 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3547471412 0 0) 0x7efeb013e490 con 0x7efeb006ab70 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 >> v1:172.21.15.33:6790/0 conn(0x7efeb0071820 legacy=0x7efeb014dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 >> v1:172.21.15.33:6789/0 conn(0x7efeb00af660 legacy=0x7efeb013ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 monclient: found mon.b 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:48.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb013f460 con 0x7efeb006ab70 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efea5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efeb7a06640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efeb7a06640 10 monclient: _renew_subs 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efeb7a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.998+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb013e0d0 con 0x7efeb006ab70 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efeb7a06640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efeb7a06640 10 monclient: _renew_subs 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efeb7a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb013da30 con 0x7efeb006ab70 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efeb7a06640 1 librados: init done 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efea00035f0 con 0x7efeb006ab70 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efea5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7efea0003980 con 0x7efeb006ab70 2024-03-29T11:35:48.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efea5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efea5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efea5ffb640 10 monclient: dump: 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:48.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:47.999+0000 7efea77fe640 4 set_mon_vals no callback set 2024-03-29T11:35:48.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.001+0000 7efeb7a06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.001+0000 7efeb7a06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.001+0000 7efeb7a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.001+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe90005080 con 0x7efeb006ab70 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.002+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7efea0003ae0 con 0x7efeb006ab70 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.003+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7efea00838b0 con 0x7efeb006ab70 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.003+0000 7efea5ffb640 10 client.4264.objecter ms_dispatch 0x7efeb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.003+0000 7efea5ffb640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.003+0000 7efea5ffb640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.003+0000 7efea5ffb640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.003+0000 7efeb4fa1640 10 monclient: get_auth_request con 0x7efe9406e000 auth_method 0 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.004+0000 7efea5ffb640 10 client.4264.objecter ms_handle_connect 0x7efe9406e000 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.010+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efea004f190 con 0x7efeb006ab70 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.010+0000 7efea5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:48.008 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.010+0000 7efea5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:48.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.420+0000 7efeb7a06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:48.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.420+0000 7efeb7a06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:48.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.420+0000 7efeb7a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:48.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.421+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe90005330 con 0x7efeb006ab70 2024-03-29T11:35:48.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.421+0000 7efea5ffb640 1 -- 172.21.15.33:0/3163375794 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7efea00510c0 con 0x7efeb006ab70 2024-03-29T11:35:48.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.421+0000 7efea5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:48.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.421+0000 7efea5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:48.420 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:48.420 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:48.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.424+0000 7efeb7a06640 10 librados: watch_flush enter 2024-03-29T11:35:48.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.424+0000 7efeb7a06640 10 librados: watch_flush exit 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 >> v1:172.21.15.153:6816/2737028074 conn(0x7efe9406e000 legacy=0x7efe940704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 10 monclient: shutdown 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 >> v1:172.21.15.153:6789/0 conn(0x7efeb006ab70 legacy=0x7efeb013d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 shutdown_connections 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 >> 172.21.15.33:0/3163375794 conn(0x7efeb00b4930 msgr2=0x7efeb00b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.425+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 shutdown_connections 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.426+0000 7efeb7a06640 1 -- 172.21.15.33:0/3163375794 wait complete. 2024-03-29T11:35:48.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:48.426+0000 7efeb7a06640 1 librados: shutdown 2024-03-29T11:35:48.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:55.245 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:55.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.403+0000 7f17cc33b640 10 monclient: get_monmap_and_config 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.403+0000 7f17cc33b640 10 monclient: build_initial_monmap 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.403+0000 7f17cc33b640 10 monclient: monmap: 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:35:55.404249+0000 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:35:55.404249+0000 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:35:55.402 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:55.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.404+0000 7f17cc33b640 1 Processor -- start 2024-03-29T11:35:55.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.404+0000 7f17cc33b640 1 -- start start 2024-03-29T11:35:55.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.404+0000 7f17cc33b640 10 monclient: init 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.405+0000 7f17cc33b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.405+0000 7f17cc33b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.405+0000 7f17cc33b640 10 monclient(hunting): picked mon.noname-c con 0x7f17c40b48a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17cc33b640 10 monclient(hunting): picked mon.noname-b con 0x7f17c40b3f70 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17cc33b640 10 monclient(hunting): picked mon.noname-a con 0x7f17c413fb20 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17cc33b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c40b43b0 con 0x7f17c413fb20 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17cc33b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c40b0b90 con 0x7f17c40b48a0 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17cc33b640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c40b19a0 con 0x7f17c40b3f70 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17cc33b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:55.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17cc33b640 10 monclient(hunting): authenticate will time out at 911.669785s 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17ca0d7640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f17c40b48a0 0x7f17c40b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:40336/0 (socket says 172.21.15.33:40336) 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17ca0d7640 1 -- 172.21.15.33:0/2610915297 learned_addr learned my addr 172.21.15.33:0/2610915297 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209475672 0 0) 0x7f17c40b0b90 con 0x7f17c40b48a0 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17c90d5640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17c90d5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17c90d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17c90d5640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.406+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac003520 con 0x7f17c40b48a0 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849531477 0 0) 0x7f17c40b19a0 con 0x7f17c40b3f70 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c40b0b90 con 0x7f17c40b3f70 2024-03-29T11:35:55.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2445545243 0 0) 0x7f17ac003520 con 0x7f17c40b48a0 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 >> v1:172.21.15.33:6790/0 conn(0x7f17c40b3f70 legacy=0x7f17c413f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 >> v1:172.21.15.33:6789/0 conn(0x7f17c413fb20 legacy=0x7f17c414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient: _finish_hunting 0 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 monclient: found mon.noname-c 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c40af800 con 0x7f17c40b48a0 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient: _finish_auth 0 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient: _check_auth_tickets 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f17b4001a10 con 0x7f17c40b48a0 2024-03-29T11:35:55.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17cc33b640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17cc33b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17c90d5640 10 monclient: dump: 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:55.412 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.407+0000 7f17cc33b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.408+0000 7f17c90d5640 1 -- 172.21.15.33:0/2610915297 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b4001b70 con 0x7f17c40b48a0 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.408+0000 7f17c90d5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.408+0000 7f17cc33b640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.408+0000 7f17cc33b640 4 set_mon_vals no callback set 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.409+0000 7f17cc33b640 10 monclient: shutdown 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.409+0000 7f17cc33b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.409+0000 7f17cc33b640 1 -- 172.21.15.33:0/2610915297 >> v1:172.21.15.153:6789/0 conn(0x7f17c40b48a0 legacy=0x7f17c40b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.409+0000 7f17cc33b640 1 -- 172.21.15.33:0/2610915297 shutdown_connections 2024-03-29T11:35:55.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.409+0000 7f17cc33b640 1 -- 172.21.15.33:0/2610915297 >> 172.21.15.33:0/2610915297 conn(0x7f17c406c8a0 msgr2=0x7f17c406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.409+0000 7f17cc33b640 1 -- 172.21.15.33:0/2610915297 shutdown_connections 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.409+0000 7f17cc33b640 1 -- 172.21.15.33:0/2610915297 wait complete. 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.410+0000 7f17cc33b640 10 monclient: build_initial_monmap 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.410+0000 7f17cc33b640 10 monclient: monmap: 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:35:55.414 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:35:55.415 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:55.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.410+0000 7f17cc33b640 1 librados: starting msgr at 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.410+0000 7f17cc33b640 1 librados: starting objecter 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.410+0000 7f17cc33b640 1 Processor -- start 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.411+0000 7f17cc33b640 1 -- start start 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.411+0000 7f17cc33b640 1 librados: setting wanted keys 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.411+0000 7f17cc33b640 1 librados: calling monclient init 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.411+0000 7f17cc33b640 10 monclient: init 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.411+0000 7f17cc33b640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 10 monclient(hunting): picked mon.noname-c con 0x7f17c40b3f70 addr v1:172.21.15.33:6790/0 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 10 monclient(hunting): picked mon.noname-b con 0x7f17c40b48a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17ca0d7640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f17c40b3f70 0x7f17c4138e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:54808/0 (socket says 172.21.15.33:54808) 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17ca0d7640 1 -- 172.21.15.33:0/366130184 learned_addr learned my addr 172.21.15.33:0/366130184 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:35:55.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 10 monclient(hunting): picked mon.noname-a con 0x7f17c413fb20 addr v1:172.21.15.33:6789/0 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17c2ffd640 10 client.?.objecter ms_handle_connect 0x7f17c40b3f70 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17c2ffd640 10 client.?.objecter ms_handle_connect 0x7f17c40b48a0 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c4146310 con 0x7f17c413fb20 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c406f730 con 0x7f17c40b48a0 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c41457b0 con 0x7f17c40b3f70 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.412+0000 7f17cc33b640 10 monclient(hunting): authenticate will time out at 911.676393s 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198846408 0 0) 0x7f17c406f730 con 0x7f17c40b48a0 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:35:55.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b0004eb0 con 0x7f17c40b48a0 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 client.?.objecter ms_handle_connect 0x7f17c413fb20 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258981446 0 0) 0x7f17c41457b0 con 0x7f17c40b3f70 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c406f730 con 0x7f17c40b3f70 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3219898902 0 0) 0x7f17b0004eb0 con 0x7f17c40b48a0 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17c41457b0 con 0x7f17c40b48a0 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f17bc001930 con 0x7f17c40b48a0 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:55.420 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.413+0000 7f17c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636995599 0 0) 0x7f17c406f730 con 0x7f17c40b3f70 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17b0004eb0 con 0x7f17c40b3f70 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f17b4004d00 con 0x7f17c40b3f70 2024-03-29T11:35:55.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 630435527 0 0) 0x7f17c41457b0 con 0x7f17c40b48a0 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 >> v1:172.21.15.33:6790/0 conn(0x7f17c40b3f70 legacy=0x7f17c4138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.422 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 >> v1:172.21.15.33:6789/0 conn(0x7f17c413fb20 legacy=0x7f17c41d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 monclient: found mon.b 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c4145940 con 0x7f17c40b48a0 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 10 monclient: _renew_subs 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c4145c00 con 0x7f17c40b48a0 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 10 monclient: _renew_subs 2024-03-29T11:35:55.423 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c4139770 con 0x7f17c40b48a0 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.414+0000 7f17cc33b640 1 librados: init done 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.415+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17bc003610 con 0x7f17c40b48a0 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.415+0000 7f17c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.415+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f17bc003920 con 0x7f17c40b48a0 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.415+0000 7f17c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.415+0000 7f17c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.415+0000 7f17c88d4640 4 set_mon_vals no callback set 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.415+0000 7f17c2ffd640 10 monclient: dump: 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:35:55.424 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.416+0000 7f17cc33b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.416+0000 7f17cc33b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.416+0000 7f17cc33b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.416+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a4005080 con 0x7f17c40b48a0 2024-03-29T11:35:55.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.419+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f17bc0198f0 con 0x7f17c40b48a0 2024-03-29T11:35:55.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.420+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f17bc083850 con 0x7f17c40b48a0 2024-03-29T11:35:55.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.420+0000 7f17c2ffd640 10 client.4270.objecter ms_dispatch 0x7f17c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:55.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.420+0000 7f17c2ffd640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:55.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.420+0000 7f17c2ffd640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:55.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.420+0000 7f17ca0d7640 10 monclient: get_auth_request con 0x7f17b006de90 auth_method 0 2024-03-29T11:35:55.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.420+0000 7f17c2ffd640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:35:55.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.420+0000 7f17c2ffd640 10 client.4270.objecter ms_handle_connect 0x7f17b006de90 2024-03-29T11:35:55.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.427+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17bc04ef60 con 0x7f17c40b48a0 2024-03-29T11:35:55.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.427+0000 7f17c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:55.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.427+0000 7f17c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:55.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.825+0000 7f17cc33b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:55.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.825+0000 7f17cc33b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:55.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.825+0000 7f17cc33b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:35:55.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.825+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a4005330 con 0x7f17c40b48a0 2024-03-29T11:35:55.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.826+0000 7f17c2ffd640 1 -- 172.21.15.33:0/366130184 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f17bc051060 con 0x7f17c40b48a0 2024-03-29T11:35:55.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.826+0000 7f17c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:55.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.826+0000 7f17c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:55.825 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:35:55.825 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:55.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 10 librados: watch_flush enter 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 10 librados: watch_flush exit 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 >> v1:172.21.15.153:6816/2737028074 conn(0x7f17b006de90 legacy=0x7f17b0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 10 monclient: shutdown 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 >> v1:172.21.15.153:6789/0 conn(0x7f17c40b48a0 legacy=0x7f17c41d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 shutdown_connections 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 >> 172.21.15.33:0/366130184 conn(0x7f17c406c8a0 msgr2=0x7f17c40b85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 shutdown_connections 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.830+0000 7f17cc33b640 1 -- 172.21.15.33:0/366130184 wait complete. 2024-03-29T11:35:55.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:35:55.831+0000 7f17cc33b640 1 librados: shutdown 2024-03-29T11:35:55.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:02.652 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:02.784 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.786+0000 7f2f39ba3640 10 monclient: get_monmap_and_config 2024-03-29T11:36:02.784 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.786+0000 7f2f39ba3640 10 monclient: build_initial_monmap 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.786+0000 7f2f39ba3640 10 monclient: monmap: 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:02.787006+0000 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:02.787006+0000 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:02.785 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.787+0000 7f2f39ba3640 1 Processor -- start 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.787+0000 7f2f39ba3640 1 -- start start 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.787+0000 7f2f39ba3640 10 monclient: init 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f2f34071de0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f2f340721c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:02.787 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f2f34145ae0 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f34145650 con 0x7f2f34145ae0 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340ae260 con 0x7f2f34071de0 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f341420d0 con 0x7f2f340721c0 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f39ba3640 10 monclient(hunting): authenticate will time out at 919.052175s 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f337fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2f34071de0 0x7f2f340aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:57580/0 (socket says 172.21.15.33:57580) 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f337fe640 1 -- 172.21.15.33:0/2372656656 learned_addr learned my addr 172.21.15.33:0/2372656656 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409189702 0 0) 0x7f2f341420d0 con 0x7f2f340721c0 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f327fc640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.788+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f1c003520 con 0x7f2f340721c0 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.789+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047554326 0 0) 0x7f2f1c003520 con 0x7f2f340721c0 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.789+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 >> v1:172.21.15.153:6789/0 conn(0x7f2f34071de0 legacy=0x7f2f340aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.788 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.789+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 >> v1:172.21.15.33:6789/0 conn(0x7f2f34145ae0 legacy=0x7f2f34145f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:02.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:02.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 1 monclient: found mon.noname-b 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340b0590 con 0x7f2f340721c0 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2f2c001a10 con 0x7f2f340721c0 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: dump: 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.791 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f39ba3640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f39ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f2c001c70 con 0x7f2f340721c0 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 1 -- 172.21.15.33:0/2372656656 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2f2c002000 con 0x7f2f340721c0 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f327fc640 10 monclient: dump: 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:02.792 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f39ba3640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.791+0000 7f2f39ba3640 4 set_mon_vals no callback set 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.792+0000 7f2f39ba3640 10 monclient: shutdown 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.792+0000 7f2f39ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.792+0000 7f2f39ba3640 1 -- 172.21.15.33:0/2372656656 >> v1:172.21.15.33:6790/0 conn(0x7f2f340721c0 legacy=0x7f2f3413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.792+0000 7f2f39ba3640 1 -- 172.21.15.33:0/2372656656 shutdown_connections 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.792+0000 7f2f39ba3640 1 -- 172.21.15.33:0/2372656656 >> 172.21.15.33:0/2372656656 conn(0x7f2f3406b660 msgr2=0x7f2f3406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.792+0000 7f2f39ba3640 1 -- 172.21.15.33:0/2372656656 shutdown_connections 2024-03-29T11:36:02.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.792+0000 7f2f39ba3640 1 -- 172.21.15.33:0/2372656656 wait complete. 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.793+0000 7f2f39ba3640 10 monclient: build_initial_monmap 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.793+0000 7f2f39ba3640 10 monclient: monmap: 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:02.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.794+0000 7f2f39ba3640 1 librados: starting msgr at 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.794+0000 7f2f39ba3640 1 librados: starting objecter 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.794+0000 7f2f39ba3640 1 Processor -- start 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.794+0000 7f2f39ba3640 1 -- start start 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.794+0000 7f2f39ba3640 1 librados: setting wanted keys 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.794+0000 7f2f39ba3640 1 librados: calling monclient init 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.794+0000 7f2f39ba3640 10 monclient: init 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f39ba3640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f39ba3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f39ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f2f34071de0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f39ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f2f34145ae0 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f39ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f2f340714c0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f39ba3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f341399e0 con 0x7f2f34145ae0 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f337fe640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f2f34071de0 0x7f2f3406f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:39200/0 (socket says 172.21.15.33:39200) 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.795+0000 7f2f337fe640 1 -- 172.21.15.33:0/1394000747 learned_addr learned my addr 172.21.15.33:0/1394000747 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:02.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 client.?.objecter ms_handle_connect 0x7f2f34071de0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3413a6c0 con 0x7f2f340714c0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3413a9a0 con 0x7f2f34071de0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f39ba3640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f39ba3640 10 monclient(hunting): authenticate will time out at 919.059989s 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581328556 0 0) 0x7f2f3413a9a0 con 0x7f2f34071de0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f20004eb0 con 0x7f2f34071de0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1760722721 0 0) 0x7f2f20004eb0 con 0x7f2f34071de0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f3413a9a0 con 0x7f2f34071de0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2f28001840 con 0x7f2f34071de0 2024-03-29T11:36:02.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): dump: 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.796+0000 7f2f1bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:02.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.797+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3004162076 0 0) 0x7f2f3413a9a0 con 0x7f2f34071de0 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.797+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 >> v1:172.21.15.153:6789/0 conn(0x7f2f340714c0 legacy=0x7f2f341e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.797+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 >> v1:172.21.15.33:6789/0 conn(0x7f2f34145ae0 legacy=0x7f2f3413b0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 1 monclient: found mon.c 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3413ac80 con 0x7f2f34071de0 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 10 client.?.objecter ms_handle_connect 0x7f2f34145ae0 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 10 monclient: _renew_subs 2024-03-29T11:36:02.803 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f34071930 con 0x7f2f34071de0 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 10 monclient: _renew_subs 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f341e7a30 con 0x7f2f34071de0 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.799+0000 7f2f39ba3640 1 librados: init done 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.800+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f280019a0 con 0x7f2f34071de0 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.800+0000 7f2f1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.800+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2f28004a40 con 0x7f2f34071de0 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.800+0000 7f2f1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.800+0000 7f2f1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.800+0000 7f2f1bfff640 10 monclient: dump: 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.804 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:02.805 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.805 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:02.805 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:02.805 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:02.805 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:02.805 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:02.805 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.800+0000 7f2f31ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:02.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.802+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f2f28004ba0 con 0x7f2f34071de0 2024-03-29T11:36:02.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f2f28084aa0 con 0x7f2f34071de0 2024-03-29T11:36:02.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f1bfff640 10 client.4295.objecter ms_dispatch 0x7f2f34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:02.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f1bfff640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:02.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f1bfff640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f32ffd640 10 monclient: get_auth_request con 0x7f2f2006dc60 auth_method 0 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f1bfff640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f39ba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f39ba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f39ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f14005080 con 0x7f2f34071de0 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.803+0000 7f2f1bfff640 10 client.4295.objecter ms_handle_connect 0x7f2f2006dc60 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.809+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f280501b0 con 0x7f2f34071de0 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.809+0000 7f2f1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:02.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:02.809+0000 7f2f1bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:03.131 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.132+0000 7f2f39ba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:03.131 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.132+0000 7f2f39ba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:03.131 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.132+0000 7f2f39ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:03.131 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.132+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f14005330 con 0x7f2f34071de0 2024-03-29T11:36:03.132 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.133+0000 7f2f1bfff640 1 -- 172.21.15.33:0/1394000747 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f2f280522b0 con 0x7f2f34071de0 2024-03-29T11:36:03.132 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.133+0000 7f2f1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:03.132 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.133+0000 7f2f1bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:03.132 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:03.132 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:03.135 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.136+0000 7f2f39ba3640 10 librados: watch_flush enter 2024-03-29T11:36:03.135 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.136+0000 7f2f39ba3640 10 librados: watch_flush exit 2024-03-29T11:36:03.135 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.136+0000 7f2f39ba3640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:36:03.135 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.136+0000 7f2f39ba3640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 >> v1:172.21.15.153:6816/2737028074 conn(0x7f2f2006dc60 legacy=0x7f2f20070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 10 monclient: shutdown 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 >> v1:172.21.15.33:6790/0 conn(0x7f2f34071de0 legacy=0x7f2f3406f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 shutdown_connections 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 >> 172.21.15.33:0/1394000747 conn(0x7f2f3406b660 msgr2=0x7f2f340b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 shutdown_connections 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 1 -- 172.21.15.33:0/1394000747 wait complete. 2024-03-29T11:36:03.136 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:03.137+0000 7f2f39ba3640 1 librados: shutdown 2024-03-29T11:36:03.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:09.956 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:10.093 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.094+0000 7f7e0c554640 10 monclient: get_monmap_and_config 2024-03-29T11:36:10.093 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.094+0000 7f7e0c554640 10 monclient: build_initial_monmap 2024-03-29T11:36:10.093 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.094+0000 7f7e0c554640 10 monclient: monmap: 2024-03-29T11:36:10.093 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:10.095387+0000 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:10.095387+0000 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:10.094 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.095+0000 7f7e0c554640 1 Processor -- start 2024-03-29T11:36:10.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.095+0000 7f7e0c554640 1 -- start start 2024-03-29T11:36:10.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.095+0000 7f7e0c554640 10 monclient: init 2024-03-29T11:36:10.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.096+0000 7f7e0c554640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:10.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.096+0000 7f7e0c554640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.096+0000 7f7e0c554640 10 monclient(hunting): picked mon.noname-a con 0x7f7e040b0770 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.096+0000 7f7e0c554640 10 monclient(hunting): picked mon.noname-b con 0x7f7e040ad850 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e0c554640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0413c1b0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e0c554640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e040ac8e0 con 0x7f7e040b0770 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e0c554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e040ad330 con 0x7f7e0413c1b0 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e0c554640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e040acfe0 con 0x7f7e040ad850 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e0c554640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e0c554640 10 monclient(hunting): authenticate will time out at 926.361068s 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e09aef640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f7e040ad850 0x7f7e040aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:50998/0 (socket says 172.21.15.33:50998) 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e09aef640 1 -- 172.21.15.33:0/2434666765 learned_addr learned my addr 172.21.15.33:0/2434666765 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.097+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377773678 0 0) 0x7f7e040acfe0 con 0x7f7e040ad850 2024-03-29T11:36:10.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 10 monclient(hunting): _negotiate 2024-03-29T11:36:10.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:10.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:10.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:36:10.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df0005730 con 0x7f7e040ad850 2024-03-29T11:36:10.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2129580253 0 0) 0x7f7df0005730 con 0x7f7e040ad850 2024-03-29T11:36:10.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 >> v1:172.21.15.153:6789/0 conn(0x7f7e0413c1b0 legacy=0x7f7e0414a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.098+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 >> v1:172.21.15.33:6789/0 conn(0x7f7e040b0770 legacy=0x7f7e04072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 10 monclient: _finish_hunting 0 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 1 monclient: found mon.noname-b 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e040ab1f0 con 0x7f7e040ad850 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 10 monclient: _finish_auth 0 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 10 monclient: _check_auth_tickets 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7df4001690 con 0x7f7e040ad850 2024-03-29T11:36:10.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.100+0000 7f7e092ee640 10 monclient: dump: 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:10.103 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e0c554640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7df4001860 con 0x7f7e040ad850 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e092ee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e092ee640 1 -- 172.21.15.33:0/2434666765 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7df4001cb0 con 0x7f7e040ad850 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e092ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e092ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e092ee640 10 monclient: dump: 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.104 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e0c554640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.101+0000 7f7e0c554640 4 set_mon_vals no callback set 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.102+0000 7f7e0c554640 10 monclient: shutdown 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.102+0000 7f7e0c554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.102+0000 7f7e0c554640 1 -- 172.21.15.33:0/2434666765 >> v1:172.21.15.33:6790/0 conn(0x7f7e040ad850 legacy=0x7f7e040aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.105 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.103+0000 7f7e0c554640 1 -- 172.21.15.33:0/2434666765 shutdown_connections 2024-03-29T11:36:10.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.103+0000 7f7e0c554640 1 -- 172.21.15.33:0/2434666765 >> 172.21.15.33:0/2434666765 conn(0x7f7e0406b660 msgr2=0x7f7e0406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:10.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.103+0000 7f7e0c554640 1 -- 172.21.15.33:0/2434666765 shutdown_connections 2024-03-29T11:36:10.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.103+0000 7f7e0c554640 1 -- 172.21.15.33:0/2434666765 wait complete. 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.104+0000 7f7e0c554640 10 monclient: build_initial_monmap 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.104+0000 7f7e0c554640 10 monclient: monmap: 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:10.112 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.104+0000 7f7e0c554640 1 librados: starting msgr at 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.104+0000 7f7e0c554640 1 librados: starting objecter 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.104+0000 7f7e0c554640 1 Processor -- start 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.105+0000 7f7e0c554640 1 -- start start 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.105+0000 7f7e0c554640 1 librados: setting wanted keys 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.105+0000 7f7e0c554640 1 librados: calling monclient init 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.105+0000 7f7e0c554640 10 monclient: init 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.106+0000 7f7e0c554640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.106+0000 7f7e0c554640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.106+0000 7f7e0c554640 10 monclient(hunting): picked mon.noname-b con 0x7f7e040ad850 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0a2f0640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7e040ad850 0x7f7e04134e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:33392/0 (socket says 172.21.15.33:33392) 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0a2f0640 1 -- 172.21.15.33:0/3923215843 learned_addr learned my addr 172.21.15.33:0/3923215843 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0c554640 10 monclient(hunting): picked mon.noname-a con 0x7f7e040b0770 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:10.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0c554640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0413c1b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e041c9890 con 0x7f7e040b0770 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e041c8720 con 0x7f7e040ad850 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e041c88b0 con 0x7f7e0413c1b0 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0c554640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.107+0000 7f7e0c554640 10 monclient(hunting): authenticate will time out at 926.371575s 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 10 client.?.objecter ms_handle_connect 0x7f7e040ad850 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794462870 0 0) 0x7f7e041c8720 con 0x7f7e040ad850 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.108+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de8004eb0 con 0x7f7e040ad850 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.109+0000 7f7dfaffd640 10 client.?.objecter ms_handle_connect 0x7f7e0413c1b0 2024-03-29T11:36:10.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.109+0000 7f7dfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.109+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891036554 0 0) 0x7f7e041c88b0 con 0x7f7e0413c1b0 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.109+0000 7f7dfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.109+0000 7f7dfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.109+0000 7f7dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.109+0000 7f7dfaffd640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e041c8720 con 0x7f7e0413c1b0 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2618233718 0 0) 0x7f7de8004eb0 con 0x7f7e040ad850 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e041c88b0 con 0x7f7e040ad850 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7e00004080 con 0x7f7e040ad850 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 10 monclient(hunting): dump: 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.115 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.110+0000 7f7dfaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 10 client.?.objecter ms_handle_connect 0x7f7e040b0770 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2264570151 0 0) 0x7f7e041c8720 con 0x7f7e0413c1b0 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7de8004eb0 con 0x7f7e0413c1b0 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7dfc001b90 con 0x7f7e0413c1b0 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 10 monclient(hunting): dump: 2024-03-29T11:36:10.116 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.111+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3316964856 0 0) 0x7f7e041c88b0 con 0x7f7e040ad850 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 >> v1:172.21.15.33:6790/0 conn(0x7f7e0413c1b0 legacy=0x7f7e041df2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 >> v1:172.21.15.33:6789/0 conn(0x7f7e040b0770 legacy=0x7f7e041debe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:10.117 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 1 monclient: found mon.b 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e041c8a40 con 0x7f7e040ad850 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e000041e0 con 0x7f7e040ad850 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7e00001180 con 0x7f7e040ad850 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.112+0000 7f7dfaffd640 10 monclient: dump: 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:10.118 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:10.119 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.113+0000 7f7e08aed640 4 set_mon_vals no callback set 2024-03-29T11:36:10.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:36:10.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 10 monclient: _renew_subs 2024-03-29T11:36:10.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e040b2140 con 0x7f7e040ad850 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 10 monclient: _renew_subs 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e04135520 con 0x7f7e040ad850 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.114+0000 7f7e0c554640 1 librados: init done 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.116+0000 7f7e0c554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.116+0000 7f7e0c554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.116+0000 7f7e0c554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.116+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e041c9a20 con 0x7f7e040ad850 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.116+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f7e00001360 con 0x7f7e040ad850 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.117+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f7e00084a00 con 0x7f7e040ad850 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.117+0000 7f7dfaffd640 10 client.4276.objecter ms_dispatch 0x7f7e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:10.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.117+0000 7f7dfaffd640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:10.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.117+0000 7f7dfaffd640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:10.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.117+0000 7f7dfaffd640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T11:36:10.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.117+0000 7f7e09aef640 10 monclient: get_auth_request con 0x7f7de806dda0 auth_method 0 2024-03-29T11:36:10.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.118+0000 7f7dfaffd640 10 client.4276.objecter ms_handle_connect 0x7f7de806dda0 2024-03-29T11:36:10.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.125+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e000502e0 con 0x7f7e040ad850 2024-03-29T11:36:10.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.125+0000 7f7dfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:10.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.125+0000 7f7dfaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:10.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.448+0000 7f7e0c554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:10.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.448+0000 7f7e0c554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:10.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.448+0000 7f7e0c554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:10.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.448+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e04075f10 con 0x7f7e040ad850 2024-03-29T11:36:10.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.449+0000 7f7dfaffd640 1 -- 172.21.15.33:0/3923215843 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f7e00052210 con 0x7f7e040ad850 2024-03-29T11:36:10.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.449+0000 7f7dfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:10.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.449+0000 7f7dfaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:10.449 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:10.449 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:10.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 10 librados: watch_flush enter 2024-03-29T11:36:10.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 10 librados: watch_flush exit 2024-03-29T11:36:10.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T11:36:10.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 10 client.4276.objecter successfully canceled tick 2024-03-29T11:36:10.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 >> v1:172.21.15.153:6816/2737028074 conn(0x7f7de806dda0 legacy=0x7f7de8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 10 monclient: shutdown 2024-03-29T11:36:10.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:10.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.453+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 >> v1:172.21.15.153:6789/0 conn(0x7f7e040ad850 legacy=0x7f7e04134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:10.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.456+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 shutdown_connections 2024-03-29T11:36:10.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.456+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 >> 172.21.15.33:0/3923215843 conn(0x7f7e0406b660 msgr2=0x7f7e0406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:10.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.457+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 shutdown_connections 2024-03-29T11:36:10.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.457+0000 7f7e0c554640 1 -- 172.21.15.33:0/3923215843 wait complete. 2024-03-29T11:36:10.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:10.457+0000 7f7e0c554640 1 librados: shutdown 2024-03-29T11:36:10.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:17.281 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:17.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.426+0000 7f266441e640 10 monclient: get_monmap_and_config 2024-03-29T11:36:17.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.426+0000 7f266441e640 10 monclient: build_initial_monmap 2024-03-29T11:36:17.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.426+0000 7f266441e640 10 monclient: monmap: 2024-03-29T11:36:17.427 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:17.427 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:17.427 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:17.427986+0000 2024-03-29T11:36:17.427 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:17.427986+0000 2024-03-29T11:36:17.428 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:17.428 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:17.428 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:17.428 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:17.428 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:17.428 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:17.428 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.429+0000 7f266441e640 1 Processor -- start 2024-03-29T11:36:17.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.429+0000 7f266441e640 1 -- start start 2024-03-29T11:36:17.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.429+0000 7f266441e640 10 monclient: init 2024-03-29T11:36:17.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 10 monclient(hunting): picked mon.noname-b con 0x7f265c0a9ac0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 10 monclient(hunting): picked mon.noname-a con 0x7f265c13ee90 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 10 monclient(hunting): picked mon.noname-c con 0x7f265c0af900 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c0afd10 con 0x7f265c13ee90 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c0af6f0 con 0x7f265c0af900 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c0aac80 con 0x7f265c0a9ac0 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:17.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.430+0000 7f266441e640 10 monclient(hunting): authenticate will time out at 933.695230s 2024-03-29T11:36:17.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.435+0000 7f26619b9640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f265c13ee90 0x7f265c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:57575/0 (socket says 172.21.15.33:57575) 2024-03-29T11:36:17.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.435+0000 7f26619b9640 1 -- 172.21.15.33:0/2561076444 learned_addr learned my addr 172.21.15.33:0/2561076444 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:17.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967551871 0 0) 0x7f265c0afd10 con 0x7f265c13ee90 2024-03-29T11:36:17.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 10 monclient(hunting): _negotiate 2024-03-29T11:36:17.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:17.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2648003520 con 0x7f265c13ee90 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439225978 0 0) 0x7f2648003520 con 0x7f265c13ee90 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 >> v1:172.21.15.33:6790/0 conn(0x7f265c0a9ac0 legacy=0x7f265c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.436+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 >> v1:172.21.15.153:6789/0 conn(0x7f265c0af900 legacy=0x7f265c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: _finish_hunting 0 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 1 monclient: found mon.noname-a 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0aaf60 con 0x7f265c13ee90 2024-03-29T11:36:17.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: _finish_auth 0 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: _check_auth_tickets 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f264c001690 con 0x7f265c13ee90 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: dump: 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:17.437 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f266441e640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f266441e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f264c001860 con 0x7f265c13ee90 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 1 -- 172.21.15.33:0/2561076444 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f264c001cb0 con 0x7f265c13ee90 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:17.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f26611b8640 10 monclient: dump: 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f266441e640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.437+0000 7f266441e640 4 set_mon_vals no callback set 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.438+0000 7f266441e640 10 monclient: shutdown 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.438+0000 7f266441e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:17.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.438+0000 7f266441e640 1 -- 172.21.15.33:0/2561076444 >> v1:172.21.15.33:6789/0 conn(0x7f265c13ee90 legacy=0x7f265c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.441 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.440+0000 7f266441e640 1 -- 172.21.15.33:0/2561076444 shutdown_connections 2024-03-29T11:36:17.441 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.440+0000 7f266441e640 1 -- 172.21.15.33:0/2561076444 >> 172.21.15.33:0/2561076444 conn(0x7f265c0b35e0 msgr2=0x7f265c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.440+0000 7f266441e640 1 -- 172.21.15.33:0/2561076444 shutdown_connections 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.440+0000 7f266441e640 1 -- 172.21.15.33:0/2561076444 wait complete. 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.441+0000 7f266441e640 10 monclient: build_initial_monmap 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.441+0000 7f266441e640 10 monclient: monmap: 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:17.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.442+0000 7f266441e640 1 librados: starting msgr at 2024-03-29T11:36:17.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.442+0000 7f266441e640 1 librados: starting objecter 2024-03-29T11:36:17.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.442+0000 7f266441e640 1 Processor -- start 2024-03-29T11:36:17.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.442+0000 7f266441e640 1 -- start start 2024-03-29T11:36:17.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.442+0000 7f266441e640 1 librados: setting wanted keys 2024-03-29T11:36:17.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.442+0000 7f266441e640 1 librados: calling monclient init 2024-03-29T11:36:17.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.442+0000 7f266441e640 10 monclient: init 2024-03-29T11:36:17.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:17.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:17.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 10 monclient(hunting): picked mon.noname-a con 0x7f265c0a9ac0 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:17.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 10 monclient(hunting): picked mon.noname-c con 0x7f265c0af900 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:17.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 10 monclient(hunting): picked mon.noname-b con 0x7f265c13ee90 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:17.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c140f00 con 0x7f265c0a9ac0 2024-03-29T11:36:17.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c13ff40 con 0x7f265c13ee90 2024-03-29T11:36:17.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c141130 con 0x7f265c0af900 2024-03-29T11:36:17.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:17.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.443+0000 7f266441e640 10 monclient(hunting): authenticate will time out at 933.708054s 2024-03-29T11:36:17.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.448+0000 7f26629bb640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f265c13ee90 0x7f265c1ddfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:35827/0 (socket says 172.21.15.33:35827) 2024-03-29T11:36:17.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.449+0000 7f26629bb640 1 -- 172.21.15.33:0/790089324 learned_addr learned my addr 172.21.15.33:0/790089324 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.451+0000 7f2652ffd640 10 client.?.objecter ms_handle_connect 0x7f265c0af900 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.451+0000 7f2652ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 client.?.objecter ms_handle_connect 0x7f265c13ee90 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 client.?.objecter ms_handle_connect 0x7f265c0a9ac0 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620758114 0 0) 0x7f265c141130 con 0x7f265c0af900 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:17.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:17.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:36:17.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2640004eb0 con 0x7f265c0af900 2024-03-29T11:36:17.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364273785 0 0) 0x7f265c13ff40 con 0x7f265c13ee90 2024-03-29T11:36:17.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:17.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:17.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:17.452 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26400051d0 con 0x7f265c13ee90 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 933886150 0 0) 0x7f2640004eb0 con 0x7f265c0af900 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2640005640 con 0x7f265c0af900 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f264c001b70 con 0x7f265c0af900 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): dump: 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:17.453 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.452+0000 7f2652ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3322483633 0 0) 0x7f26400051d0 con 0x7f265c13ee90 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2640004eb0 con 0x7f265c13ee90 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677140280 0 0) 0x7f265c140f00 con 0x7f265c0a9ac0 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:36:17.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26400051d0 con 0x7f265c0a9ac0 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2654001ba0 con 0x7f265c13ee90 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient(hunting): dump: 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3621753237 0 0) 0x7f2640005640 con 0x7f265c0af900 2024-03-29T11:36:17.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 >> v1:172.21.15.153:6789/0 conn(0x7f265c13ee90 legacy=0x7f265c1ddfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 >> v1:172.21.15.33:6789/0 conn(0x7f265c0a9ac0 legacy=0x7f265c13f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 monclient: found mon.c 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c141410 con 0x7f265c0af900 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f2652ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 10 monclient: _renew_subs 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c1416f0 con 0x7f265c0af900 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 10 monclient: _renew_subs 2024-03-29T11:36:17.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:17.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c137690 con 0x7f265c0af900 2024-03-29T11:36:17.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.453+0000 7f266441e640 1 librados: init done 2024-03-29T11:36:17.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.455+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f264c004a90 con 0x7f265c0af900 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.455+0000 7f2652ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.455+0000 7f26609b7640 4 set_mon_vals no callback set 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.456+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f264c001250 con 0x7f265c0af900 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.456+0000 7f2652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.456+0000 7f2652ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.456+0000 7f2652ffd640 10 monclient: dump: 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:17.459 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.456+0000 7f266441e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.457+0000 7f266441e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.457+0000 7f266441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.457+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2644005080 con 0x7f265c0af900 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.457+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f264c01ac10 con 0x7f265c0af900 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.458+0000 7f26621ba640 10 monclient: get_auth_request con 0x7f264006e4e0 auth_method 0 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.458+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f264c054570 con 0x7f265c0af900 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.458+0000 7f2652ffd640 10 client.4307.objecter ms_dispatch 0x7f265c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.458+0000 7f2652ffd640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.458+0000 7f2652ffd640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.458+0000 7f2652ffd640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-29T11:36:17.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.458+0000 7f2652ffd640 10 client.4307.objecter ms_handle_connect 0x7f264006e4e0 2024-03-29T11:36:17.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.468+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f264c050810 con 0x7f265c0af900 2024-03-29T11:36:17.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.468+0000 7f2652ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:17.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.468+0000 7f2652ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:17.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.801+0000 7f266441e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.801+0000 7f266441e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.801+0000 7f266441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:17.801 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.801+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2644005330 con 0x7f265c0af900 2024-03-29T11:36:17.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.802+0000 7f2652ffd640 1 -- 172.21.15.33:0/790089324 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f264c050810 con 0x7f265c0af900 2024-03-29T11:36:17.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.802+0000 7f2652ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.802+0000 7f2652ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:17.803 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:17.803 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:17.805 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.805+0000 7f266441e640 10 librados: watch_flush enter 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 10 librados: watch_flush exit 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 10 client.4307.objecter successfully canceled tick 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 >> v1:172.21.15.153:6816/2737028074 conn(0x7f264006e4e0 legacy=0x7f26400709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 10 monclient: shutdown 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 >> v1:172.21.15.33:6790/0 conn(0x7f265c0af900 legacy=0x7f265c141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 shutdown_connections 2024-03-29T11:36:17.806 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.806+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 >> 172.21.15.33:0/790089324 conn(0x7f265c0b35e0 msgr2=0x7f265c13b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.807+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 shutdown_connections 2024-03-29T11:36:17.807 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.808+0000 7f266441e640 1 -- 172.21.15.33:0/790089324 wait complete. 2024-03-29T11:36:17.809 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:17.810+0000 7f266441e640 1 librados: shutdown 2024-03-29T11:36:17.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:24.633 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:24.784 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.784+0000 7f048659d640 10 monclient: get_monmap_and_config 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.784+0000 7f048659d640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.784+0000 7f048659d640 10 monclient: monmap: 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:24.786542+0000 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:24.786542+0000 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:24.785 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:24.786 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.786+0000 7f048659d640 1 Processor -- start 2024-03-29T11:36:24.786 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.786+0000 7f048659d640 1 -- start start 2024-03-29T11:36:24.786 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.786+0000 7f048659d640 10 monclient: init 2024-03-29T11:36:24.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:24.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 10 monclient(hunting): picked mon.noname-c con 0x7f04800b0860 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:24.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 10 monclient(hunting): picked mon.noname-b con 0x7f048013ef30 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:24.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 10 monclient(hunting): picked mon.noname-a con 0x7f04800af8e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:24.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800b0c40 con 0x7f04800af8e0 2024-03-29T11:36:24.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800aed10 con 0x7f04800b0860 2024-03-29T11:36:24.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800afd50 con 0x7f048013ef30 2024-03-29T11:36:24.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.787+0000 7f048659d640 10 monclient(hunting): authenticate will time out at 941.051893s 2024-03-29T11:36:24.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.792+0000 7f0484b3a640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f04800af8e0 0x7f0480149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:42087/0 (socket says 172.21.15.33:42087) 2024-03-29T11:36:24.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.792+0000 7f0484b3a640 1 -- 172.21.15.33:0/3270847001 learned_addr learned my addr 172.21.15.33:0/3270847001 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:24.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.793+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903069306 0 0) 0x7f04800b0c40 con 0x7f04800af8e0 2024-03-29T11:36:24.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.793+0000 7f047effd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.793+0000 7f047effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.793+0000 7f047effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.793+0000 7f047effd640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:36:24.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.793+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0468003520 con 0x7f04800af8e0 2024-03-29T11:36:24.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3562875313 0 0) 0x7f0468003520 con 0x7f04800af8e0 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 >> v1:172.21.15.33:6790/0 conn(0x7f048013ef30 legacy=0x7f04800af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 >> v1:172.21.15.153:6789/0 conn(0x7f04800b0860 legacy=0x7f048013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 monclient: found mon.noname-a 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04800ab1b0 con 0x7f04800af8e0 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0478001a10 con 0x7f04800af8e0 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:24.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: dump: 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0478001be0 con 0x7f04800af8e0 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 1 -- 172.21.15.33:0/3270847001 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0478002030 con 0x7f04800af8e0 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f047effd640 10 monclient: dump: 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:24.796 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f048659d640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f048659d640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.794+0000 7f048659d640 4 set_mon_vals no callback set 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.795+0000 7f048659d640 10 monclient: shutdown 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.795+0000 7f048659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.795+0000 7f048659d640 1 -- 172.21.15.33:0/3270847001 >> v1:172.21.15.33:6789/0 conn(0x7f04800af8e0 legacy=0x7f0480149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.795+0000 7f048659d640 1 -- 172.21.15.33:0/3270847001 shutdown_connections 2024-03-29T11:36:24.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.795+0000 7f048659d640 1 -- 172.21.15.33:0/3270847001 >> 172.21.15.33:0/3270847001 conn(0x7f048006b660 msgr2=0x7f048006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.797+0000 7f048659d640 1 -- 172.21.15.33:0/3270847001 shutdown_connections 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.798+0000 7f048659d640 1 -- 172.21.15.33:0/3270847001 wait complete. 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.798+0000 7f048659d640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.798+0000 7f048659d640 10 monclient: monmap: 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.798 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:24.799 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:24.799 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:24.799 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:24.799 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:24.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.799+0000 7f048659d640 1 librados: starting msgr at 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.799+0000 7f048659d640 1 librados: starting objecter 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.799+0000 7f048659d640 1 Processor -- start 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.799+0000 7f048659d640 1 -- start start 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.799+0000 7f048659d640 1 librados: setting wanted keys 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.799+0000 7f048659d640 1 librados: calling monclient init 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.799+0000 7f048659d640 10 monclient: init 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.800+0000 7f048659d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.800 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.800+0000 7f048659d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 10 monclient(hunting): picked mon.noname-a con 0x7f04800af8e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 10 monclient(hunting): picked mon.noname-b con 0x7f04800b0860 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 10 monclient(hunting): picked mon.noname-c con 0x7f048013ef30 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800b2c90 con 0x7f04800af8e0 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800b2960 con 0x7f04800b0860 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800b3c60 con 0x7f048013ef30 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.802 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.802+0000 7f048659d640 10 monclient(hunting): authenticate will time out at 941.067499s 2024-03-29T11:36:24.805 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.806+0000 7f047ffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f04800af8e0 0x7f04800b2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:42199/0 (socket says 172.21.15.33:42199) 2024-03-29T11:36:24.805 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.806+0000 7f047ffff640 1 -- 172.21.15.33:0/3285762778 learned_addr learned my addr 172.21.15.33:0/3285762778 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:24.813 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 client.?.objecter ms_handle_connect 0x7f04800b0860 2024-03-29T11:36:24.813 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.813 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 client.?.objecter ms_handle_connect 0x7f04800af8e0 2024-03-29T11:36:24.813 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.813 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 client.?.objecter ms_handle_connect 0x7f048013ef30 2024-03-29T11:36:24.813 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374120956 0 0) 0x7f04800b3c60 con 0x7f048013ef30 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c004eb0 con 0x7f048013ef30 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907908782 0 0) 0x7f04800b2c90 con 0x7f04800af8e0 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.809+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04800b3c60 con 0x7f04800af8e0 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693181337 0 0) 0x7f04800b2960 con 0x7f04800b0860 2024-03-29T11:36:24.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04800b2c90 con 0x7f04800b0860 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2993976127 0 0) 0x7f046c004eb0 con 0x7f048013ef30 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04800b2960 con 0x7f048013ef30 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0478001ef0 con 0x7f048013ef30 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.810+0000 7f047cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 10 monclient(hunting): dump: 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.815 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4232025959 0 0) 0x7f04800b3c60 con 0x7f04800af8e0 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f046c004eb0 con 0x7f04800af8e0 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f04740018e0 con 0x7f04800af8e0 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 10 monclient(hunting): dump: 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:24.816 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 174430102 0 0) 0x7f04800b2c90 con 0x7f04800b0860 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04800b3c60 con 0x7f04800b0860 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604736919 0 0) 0x7f04800b2960 con 0x7f048013ef30 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 >> v1:172.21.15.153:6789/0 conn(0x7f04800b0860 legacy=0x7f0480132660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.811+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 >> v1:172.21.15.33:6789/0 conn(0x7f04800af8e0 legacy=0x7f04800b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 1 monclient: found mon.c 2024-03-29T11:36:24.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04800b3f40 con 0x7f048013ef30 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f047cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f048659d640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.812+0000 7f048659d640 10 monclient: _renew_subs 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.813+0000 7f048659d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.813+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0480071a10 con 0x7f048013ef30 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.813+0000 7f048659d640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.813+0000 7f048659d640 10 monclient: _renew_subs 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.813+0000 7f048659d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.813+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0480132f50 con 0x7f048013ef30 2024-03-29T11:36:24.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.813+0000 7f048659d640 1 librados: init done 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0478001390 con 0x7f048013ef30 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f047cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f04780015d0 con 0x7f048013ef30 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f047cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f047cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f047cff9640 10 monclient: dump: 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:24.819 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f048659d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f048659d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f048659d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.815+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0480075f10 con 0x7f048013ef30 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.816+0000 7f047dffb640 4 set_mon_vals no callback set 2024-03-29T11:36:24.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.817+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f047801af50 con 0x7f048013ef30 2024-03-29T11:36:24.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.818+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f0478084d30 con 0x7f048013ef30 2024-03-29T11:36:24.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.818+0000 7f047cff9640 10 client.4310.objecter ms_dispatch 0x7f0480000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:24.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.818+0000 7f047cff9640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:24.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.818+0000 7f047cff9640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:24.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.818+0000 7f047cff9640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T11:36:24.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.819+0000 7f047ffff640 10 monclient: get_auth_request con 0x7f046c06e090 auth_method 0 2024-03-29T11:36:24.821 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.819+0000 7f047cff9640 10 client.4310.objecter ms_handle_connect 0x7f046c06e090 2024-03-29T11:36:24.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.825+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0478050440 con 0x7f048013ef30 2024-03-29T11:36:24.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.825+0000 7f047cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.825 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:24.825+0000 7f047cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:25.161 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.160+0000 7f048659d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:25.161 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.160+0000 7f048659d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:25.161 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.160+0000 7f048659d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:25.161 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.160+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0480072330 con 0x7f048013ef30 2024-03-29T11:36:25.162 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.162+0000 7f047cff9640 1 -- 172.21.15.33:0/3285762778 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f0478052540 con 0x7f048013ef30 2024-03-29T11:36:25.162 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.162+0000 7f047cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:25.162 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.162+0000 7f047cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:25.163 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:25.163 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 10 librados: watch_flush enter 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 10 librados: watch_flush exit 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 10 client.4310.objecter successfully canceled tick 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 >> v1:172.21.15.153:6816/2737028074 conn(0x7f046c06e090 legacy=0x7f046c070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 10 monclient: shutdown 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 >> v1:172.21.15.33:6790/0 conn(0x7f048013ef30 legacy=0x7f04801d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 shutdown_connections 2024-03-29T11:36:25.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 >> 172.21.15.33:0/3285762778 conn(0x7f048006b660 msgr2=0x7f048013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 shutdown_connections 2024-03-29T11:36:25.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 1 -- 172.21.15.33:0/3285762778 wait complete. 2024-03-29T11:36:25.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:25.170+0000 7f048659d640 1 librados: shutdown 2024-03-29T11:36:25.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:31.995 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.152+0000 7f96337c9640 10 monclient: get_monmap_and_config 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.152+0000 7f96337c9640 10 monclient: build_initial_monmap 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.152+0000 7f96337c9640 10 monclient: monmap: 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:32.154054+0000 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:32.154054+0000 2024-03-29T11:36:32.152 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.153+0000 7f96337c9640 1 Processor -- start 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.153+0000 7f96337c9640 1 -- start start 2024-03-29T11:36:32.153 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.153+0000 7f96337c9640 10 monclient: init 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 10 monclient(hunting): picked mon.noname-c con 0x7f962c1457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 10 monclient(hunting): picked mon.noname-b con 0x7f962c13f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 10 monclient(hunting): picked mon.noname-a con 0x7f962c0a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c0aaef0 con 0x7f962c0a9d70 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c145bd0 con 0x7f962c1457f0 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c145eb0 con 0x7f962c13f650 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:32.154 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.154+0000 7f96337c9640 10 monclient(hunting): authenticate will time out at 948.419319s 2024-03-29T11:36:32.158 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.159+0000 7f9631565640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f962c1457f0 0x7f962c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:40047/0 (socket says 172.21.15.33:40047) 2024-03-29T11:36:32.159 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.159+0000 7f9631565640 1 -- 172.21.15.33:0/1695202523 learned_addr learned my addr 172.21.15.33:0/1695202523 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:32.162 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664959741 0 0) 0x7f962c145bd0 con 0x7f962c1457f0 2024-03-29T11:36:32.162 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:32.162 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:32.162 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9618003520 con 0x7f962c1457f0 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880910560 0 0) 0x7f962c145eb0 con 0x7f962c13f650 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:36:32.163 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962c145bd0 con 0x7f962c13f650 2024-03-29T11:36:32.164 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615524606 0 0) 0x7f9618003520 con 0x7f962c1457f0 2024-03-29T11:36:32.164 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 >> v1:172.21.15.33:6790/0 conn(0x7f962c13f650 legacy=0x7f962c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 >> v1:172.21.15.33:6789/0 conn(0x7f962c0a9d70 legacy=0x7f962c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 monclient: found mon.noname-c 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c146340 con 0x7f962c1457f0 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f961c001690 con 0x7f962c1457f0 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 10 monclient: dump: 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:32.165 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f96337c9640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f96337c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.163+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f961c0017f0 con 0x7f962c1457f0 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f9623fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:32.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f96337c9640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f96337c9640 4 set_mon_vals no callback set 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f9623fff640 1 -- 172.21.15.33:0/1695202523 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f961c001c00 con 0x7f962c1457f0 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f9623fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f9623fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f9623fff640 10 monclient: dump: 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:32.167 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:32.168 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f96337c9640 10 monclient: shutdown 2024-03-29T11:36:32.168 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f96337c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:32.168 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.164+0000 7f96337c9640 1 -- 172.21.15.33:0/1695202523 >> v1:172.21.15.153:6789/0 conn(0x7f962c1457f0 legacy=0x7f962c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.168 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.165+0000 7f96337c9640 1 -- 172.21.15.33:0/1695202523 shutdown_connections 2024-03-29T11:36:32.168 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.165+0000 7f96337c9640 1 -- 172.21.15.33:0/1695202523 >> 172.21.15.33:0/1695202523 conn(0x7f962c066a00 msgr2=0x7f962c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:32.168 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.166+0000 7f96337c9640 1 -- 172.21.15.33:0/1695202523 shutdown_connections 2024-03-29T11:36:32.170 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.170+0000 7f96337c9640 1 -- 172.21.15.33:0/1695202523 wait complete. 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.170+0000 7f96337c9640 10 monclient: build_initial_monmap 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.170+0000 7f96337c9640 10 monclient: monmap: 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.171+0000 7f96337c9640 1 librados: starting msgr at 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.171+0000 7f96337c9640 1 librados: starting objecter 2024-03-29T11:36:32.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.171+0000 7f96337c9640 1 Processor -- start 2024-03-29T11:36:32.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.171+0000 7f96337c9640 1 -- start start 2024-03-29T11:36:32.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.171+0000 7f96337c9640 1 librados: setting wanted keys 2024-03-29T11:36:32.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.171+0000 7f96337c9640 1 librados: calling monclient init 2024-03-29T11:36:32.173 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.171+0000 7f96337c9640 10 monclient: init 2024-03-29T11:36:32.173 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.172+0000 7f96337c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:32.173 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.172+0000 7f96337c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:32.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.176+0000 7f96337c9640 10 monclient(hunting): picked mon.noname-b con 0x7f962c0a9d70 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:32.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.176+0000 7f9631565640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f962c0a9d70 0x7f962c13d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:40183/0 (socket says 172.21.15.33:40183) 2024-03-29T11:36:32.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f96337c9640 10 monclient(hunting): picked mon.noname-c con 0x7f962c13f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:32.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f96337c9640 10 monclient(hunting): picked mon.noname-a con 0x7f962c1457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:32.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f96337c9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c13fda0 con 0x7f962c1457f0 2024-03-29T11:36:32.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f96337c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c142c60 con 0x7f962c0a9d70 2024-03-29T11:36:32.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f96337c9640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c142a20 con 0x7f962c13f650 2024-03-29T11:36:32.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f96337c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:32.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f96337c9640 10 monclient(hunting): authenticate will time out at 948.444026s 2024-03-29T11:36:32.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.179+0000 7f9631565640 1 -- 172.21.15.33:0/3815649645 learned_addr learned my addr 172.21.15.33:0/3815649645 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:32.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 client.?.objecter ms_handle_connect 0x7f962c0a9d70 2024-03-29T11:36:32.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:32.181 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 client.?.objecter ms_handle_connect 0x7f962c13f650 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656581883 0 0) 0x7f962c142c60 con 0x7f962c0a9d70 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610004eb0 con 0x7f962c0a9d70 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578185895 0 0) 0x7f962c142a20 con 0x7f962c13f650 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:32.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:32.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:32.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:36:32.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.182+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962c142c60 con 0x7f962c13f650 2024-03-29T11:36:32.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.183+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1009096502 0 0) 0x7f9610004eb0 con 0x7f962c0a9d70 2024-03-29T11:36:32.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.183+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f962c142a20 con 0x7f962c0a9d70 2024-03-29T11:36:32.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.183+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773147339 0 0) 0x7f962c142c60 con 0x7f962c13f650 2024-03-29T11:36:32.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.183+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9610004eb0 con 0x7f962c13f650 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.183+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f961c001690 con 0x7f962c0a9d70 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:32.184 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f96280017c0 con 0x7f962c13f650 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:32.185 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 client.?.objecter ms_handle_connect 0x7f962c1457f0 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3859462964 0 0) 0x7f962c142a20 con 0x7f962c0a9d70 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 >> v1:172.21.15.33:6790/0 conn(0x7f962c13f650 legacy=0x7f962c141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 >> v1:172.21.15.33:6789/0 conn(0x7f962c1457f0 legacy=0x7f962c141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 1 monclient: found mon.b 2024-03-29T11:36:32.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.184+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c140f20 con 0x7f962c0a9d70 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 10 monclient: _renew_subs 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c141200 con 0x7f962c0a9d70 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 10 monclient: _renew_subs 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f962c142690 con 0x7f962c0a9d70 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f96337c9640 1 librados: init done 2024-03-29T11:36:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f961c004a90 con 0x7f962c0a9d70 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f961c004ee0 con 0x7f962c0a9d70 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.185+0000 7f9621ffb640 10 monclient: dump: 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:32.188 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:32.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.186+0000 7f96237fe640 4 set_mon_vals no callback set 2024-03-29T11:36:32.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.192+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f961c01aeb0 con 0x7f962c0a9d70 2024-03-29T11:36:32.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.193+0000 7f9630d64640 10 monclient: get_auth_request con 0x7f961006df30 auth_method 0 2024-03-29T11:36:32.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.193+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f961c084b80 con 0x7f962c0a9d70 2024-03-29T11:36:32.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.193+0000 7f9621ffb640 10 client.4291.objecter ms_dispatch 0x7f962c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.193+0000 7f9621ffb640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.193+0000 7f9621ffb640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.194+0000 7f9621ffb640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.194+0000 7f9621ffb640 10 client.4291.objecter ms_handle_connect 0x7f961006df30 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.194+0000 7f96337c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.194+0000 7f96337c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.194+0000 7f96337c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.194+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c005080 con 0x7f962c0a9d70 2024-03-29T11:36:32.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.203+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f961c050460 con 0x7f962c0a9d70 2024-03-29T11:36:32.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.203+0000 7f9621ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.203+0000 7f9621ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:32.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.535+0000 7f96337c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:32.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.535+0000 7f96337c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:32.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.535+0000 7f96337c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:36:32.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.535+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f960c005330 con 0x7f962c0a9d70 2024-03-29T11:36:32.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.536+0000 7f9621ffb640 1 -- 172.21.15.33:0/3815649645 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f961c052390 con 0x7f962c0a9d70 2024-03-29T11:36:32.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.536+0000 7f9621ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:32.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.536+0000 7f9621ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:32.536 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:32.537 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:32.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.540+0000 7f96337c9640 10 librados: watch_flush enter 2024-03-29T11:36:32.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.540+0000 7f96337c9640 10 librados: watch_flush exit 2024-03-29T11:36:32.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.540+0000 7f96337c9640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T11:36:32.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.540+0000 7f96337c9640 10 client.4291.objecter successfully canceled tick 2024-03-29T11:36:32.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.541+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 >> v1:172.21.15.153:6816/2737028074 conn(0x7f961006df30 legacy=0x7f96100703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.541+0000 7f96337c9640 10 monclient: shutdown 2024-03-29T11:36:32.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.541+0000 7f96337c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:32.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.541+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 >> v1:172.21.15.153:6789/0 conn(0x7f962c0a9d70 legacy=0x7f962c13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:32.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.541+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 shutdown_connections 2024-03-29T11:36:32.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.541+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 >> 172.21.15.33:0/3815649645 conn(0x7f962c066a00 msgr2=0x7f962c1dacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:32.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.544+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 shutdown_connections 2024-03-29T11:36:32.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.544+0000 7f96337c9640 1 -- 172.21.15.33:0/3815649645 wait complete. 2024-03-29T11:36:32.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:32.544+0000 7f96337c9640 1 librados: shutdown 2024-03-29T11:36:32.561 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:39.368 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:39.515 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.517+0000 7f7f973ca640 10 monclient: get_monmap_and_config 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.517+0000 7f7f973ca640 10 monclient: build_initial_monmap 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.517+0000 7f7f973ca640 10 monclient: monmap: 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:39.517818+0000 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:39.517818+0000 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:39.516 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:39.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.518+0000 7f7f973ca640 1 Processor -- start 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.518+0000 7f7f973ca640 1 -- start start 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.518+0000 7f7f973ca640 10 monclient: init 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 10 monclient(hunting): picked mon.noname-b con 0x7f7f901422b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 10 monclient(hunting): picked mon.noname-a con 0x7f7f90142690 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 10 monclient(hunting): picked mon.noname-c con 0x7f7f900b17d0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f90142c80 con 0x7f7f90142690 2024-03-29T11:36:39.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f900aae40 con 0x7f7f900b17d0 2024-03-29T11:36:39.521 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9013fbd0 con 0x7f7f901422b0 2024-03-29T11:36:39.521 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:39.521 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.519+0000 7f7f973ca640 10 monclient(hunting): authenticate will time out at 955.783133s 2024-03-29T11:36:39.521 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.523+0000 7f7f95967640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7f900b17d0 0x7f7f900b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:36335/0 (socket says 172.21.15.33:36335) 2024-03-29T11:36:39.521 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.523+0000 7f7f95967640 1 -- 172.21.15.33:0/1519800164 learned_addr learned my addr 172.21.15.33:0/1519800164 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:39.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309976505 0 0) 0x7f7f90142c80 con 0x7f7f90142690 2024-03-29T11:36:39.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:36:39.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c003520 con 0x7f7f90142690 2024-03-29T11:36:39.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140571800 0 0) 0x7f7f9013fbd0 con 0x7f7f901422b0 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.524+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f90142c80 con 0x7f7f901422b0 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2972881438 0 0) 0x7f7f7c003520 con 0x7f7f90142690 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 >> v1:172.21.15.33:6790/0 conn(0x7f7f901422b0 legacy=0x7f7f9013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 >> v1:172.21.15.153:6789/0 conn(0x7f7f900b17d0 legacy=0x7f7f900b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 1 monclient: found mon.noname-a 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f90151940 con 0x7f7f90142690 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:39.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7f80001690 con 0x7f7f90142690 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.525+0000 7f7f87fff640 10 monclient: dump: 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:39.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.526+0000 7f7f973ca640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:36:39.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.526+0000 7f7f973ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.528+0000 7f7f87fff640 1 -- 172.21.15.33:0/1519800164 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f800017f0 con 0x7f7f90142690 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.528+0000 7f7f87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.528+0000 7f7f973ca640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.528+0000 7f7f973ca640 4 set_mon_vals no callback set 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.529+0000 7f7f973ca640 10 monclient: shutdown 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.529+0000 7f7f973ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.529+0000 7f7f973ca640 1 -- 172.21.15.33:0/1519800164 >> v1:172.21.15.33:6789/0 conn(0x7f7f90142690 legacy=0x7f7f900b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.530+0000 7f7f973ca640 1 -- 172.21.15.33:0/1519800164 shutdown_connections 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.530+0000 7f7f973ca640 1 -- 172.21.15.33:0/1519800164 >> 172.21.15.33:0/1519800164 conn(0x7f7f90066a00 msgr2=0x7f7f90066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:39.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.530+0000 7f7f973ca640 1 -- 172.21.15.33:0/1519800164 shutdown_connections 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.530+0000 7f7f973ca640 1 -- 172.21.15.33:0/1519800164 wait complete. 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.531+0000 7f7f973ca640 10 monclient: build_initial_monmap 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.531+0000 7f7f973ca640 10 monclient: monmap: 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:39.530 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:39.531 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:39.531 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:39.531 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:39.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.532+0000 7f7f973ca640 1 librados: starting msgr at 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.532+0000 7f7f973ca640 1 librados: starting objecter 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.532+0000 7f7f973ca640 1 Processor -- start 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.532+0000 7f7f973ca640 1 -- start start 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.532+0000 7f7f973ca640 1 librados: setting wanted keys 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.532+0000 7f7f973ca640 1 librados: calling monclient init 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.532+0000 7f7f973ca640 10 monclient: init 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 10 monclient(hunting): picked mon.noname-c con 0x7f7f900b17d0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 10 monclient(hunting): picked mon.noname-b con 0x7f7f901422b0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 10 monclient(hunting): picked mon.noname-a con 0x7f7f90142690 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9012d660 con 0x7f7f90142690 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f901db0a0 con 0x7f7f901422b0 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f90141d20 con 0x7f7f900b17d0 2024-03-29T11:36:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:39.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.533+0000 7f7f973ca640 10 monclient(hunting): authenticate will time out at 955.796856s 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.540+0000 7f7f95166640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f7f900b17d0 0x7f7f901da990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:60593/0 (socket says 172.21.15.33:60593) 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.540+0000 7f7f95166640 1 -- 172.21.15.33:0/2844274184 learned_addr learned my addr 172.21.15.33:0/2844274184 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.540+0000 7f7f85ffb640 10 client.?.objecter ms_handle_connect 0x7f7f900b17d0 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.540+0000 7f7f85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.541+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575911201 0 0) 0x7f7f90141d20 con 0x7f7f900b17d0 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.541+0000 7f7f85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.541+0000 7f7f85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.541+0000 7f7f85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.541+0000 7f7f85ffb640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:36:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.541+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f74004eb0 con 0x7f7f900b17d0 2024-03-29T11:36:39.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 client.?.objecter ms_handle_connect 0x7f7f901422b0 2024-03-29T11:36:39.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:39.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930316639 0 0) 0x7f7f901db0a0 con 0x7f7f901422b0 2024-03-29T11:36:39.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:39.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:39.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:39.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f90141d20 con 0x7f7f901422b0 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3710522632 0 0) 0x7f7f74004eb0 con 0x7f7f900b17d0 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f901db0a0 con 0x7f7f900b17d0 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7f88001e10 con 0x7f7f900b17d0 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:39.546 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.545+0000 7f7f85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4088752212 0 0) 0x7f7f901db0a0 con 0x7f7f900b17d0 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 >> v1:172.21.15.153:6789/0 conn(0x7f7f901422b0 legacy=0x7f7f9012c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 >> v1:172.21.15.33:6789/0 conn(0x7f7f90142690 legacy=0x7f7f9012cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 1 monclient: found mon.c 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f900b6080 con 0x7f7f900b17d0 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f973ca640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f973ca640 10 monclient: _renew_subs 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f973ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.546+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f900b7630 con 0x7f7f900b17d0 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f973ca640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f973ca640 10 monclient: _renew_subs 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f973ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f901415f0 con 0x7f7f900b17d0 2024-03-29T11:36:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f973ca640 1 librados: init done 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f88002050 con 0x7f7f900b17d0 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7f88003de0 con 0x7f7f900b17d0 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f877fe640 4 set_mon_vals no callback set 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.547+0000 7f7f85ffb640 10 monclient: dump: 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:39.549 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.548+0000 7f7f973ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.548+0000 7f7f973ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.548+0000 7f7f973ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.548+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f70005080 con 0x7f7f900b17d0 2024-03-29T11:36:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.552+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f7f88003f40 con 0x7f7f900b17d0 2024-03-29T11:36:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.552+0000 7f7f94965640 10 monclient: get_auth_request con 0x7f7f7406dd30 auth_method 0 2024-03-29T11:36:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.553+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f7f88084e50 con 0x7f7f900b17d0 2024-03-29T11:36:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.553+0000 7f7f85ffb640 10 client.4322.objecter ms_dispatch 0x7f7f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.553+0000 7f7f85ffb640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.553+0000 7f7f85ffb640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.553+0000 7f7f85ffb640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:36:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.553+0000 7f7f85ffb640 10 client.4322.objecter ms_handle_connect 0x7f7f7406dd30 2024-03-29T11:36:39.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.557+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f88050730 con 0x7f7f900b17d0 2024-03-29T11:36:39.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.557+0000 7f7f85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:39.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.557+0000 7f7f85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:39.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.886+0000 7f7f973ca640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.886+0000 7f7f973ca640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.886+0000 7f7f973ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:36:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.886+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f70005330 con 0x7f7f900b17d0 2024-03-29T11:36:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.890+0000 7f7f85ffb640 1 -- 172.21.15.33:0/2844274184 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f7f88052660 con 0x7f7f900b17d0 2024-03-29T11:36:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.890+0000 7f7f85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.890+0000 7f7f85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:39.890 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:39.890 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:39.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.894+0000 7f7f973ca640 10 librados: watch_flush enter 2024-03-29T11:36:39.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.894+0000 7f7f973ca640 10 librados: watch_flush exit 2024-03-29T11:36:39.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.894+0000 7f7f973ca640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:36:39.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.894+0000 7f7f973ca640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:36:39.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.895+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 >> v1:172.21.15.153:6816/2737028074 conn(0x7f7f7406dd30 legacy=0x7f7f740701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.895+0000 7f7f973ca640 10 monclient: shutdown 2024-03-29T11:36:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.895+0000 7f7f973ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.895+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 >> v1:172.21.15.33:6790/0 conn(0x7f7f900b17d0 legacy=0x7f7f901da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.896+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 shutdown_connections 2024-03-29T11:36:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.896+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 >> 172.21.15.33:0/2844274184 conn(0x7f7f90066a00 msgr2=0x7f7f9013d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:39.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.899+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 shutdown_connections 2024-03-29T11:36:39.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.900+0000 7f7f973ca640 1 -- 172.21.15.33:0/2844274184 wait complete. 2024-03-29T11:36:39.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:39.900+0000 7f7f973ca640 1 librados: shutdown 2024-03-29T11:36:39.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:46.722 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.876+0000 7f1ee093e640 10 monclient: get_monmap_and_config 2024-03-29T11:36:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.876+0000 7f1ee093e640 10 monclient: build_initial_monmap 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.876+0000 7f1ee093e640 10 monclient: monmap: 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:46.877547+0000 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:46.877547+0000 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:46.876 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:46.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.881+0000 7f1ee093e640 1 Processor -- start 2024-03-29T11:36:46.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.883+0000 7f1ee093e640 1 -- start start 2024-03-29T11:36:46.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.883+0000 7f1ee093e640 10 monclient: init 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 10 monclient(hunting): picked mon.noname-a con 0x7f1edc1422b0 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 10 monclient(hunting): picked mon.noname-c con 0x7f1edc142690 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 10 monclient(hunting): picked mon.noname-b con 0x7f1edc0b17d0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc142c80 con 0x7f1edc1422b0 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0aae40 con 0x7f1edc142690 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc13fbd0 con 0x7f1edc0b17d0 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.884+0000 7f1ee093e640 10 monclient(hunting): authenticate will time out at 963.148031s 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.888+0000 7f1ed9d9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1edc142690 0x7f1edc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:59479/0 (socket says 172.21.15.33:59479) 2024-03-29T11:36:46.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.888+0000 7f1ed9d9b640 1 -- 172.21.15.33:0/3812320016 learned_addr learned my addr 172.21.15.33:0/3812320016 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:46.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.888+0000 7f1edad9d640 1 -- 172.21.15.33:0/3812320016 >> v1:172.21.15.33:6790/0 conn(0x7f1edc0b17d0 legacy=0x7f1edc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:36:46.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.888+0000 7f1edad9d640 1 -- 172.21.15.33:0/3812320016 >> v1:172.21.15.33:6790/0 conn(0x7f1edc0b17d0 legacy=0x7f1edc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:36:46.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.888+0000 7f1edad9d640 1 --1- 172.21.15.33:0/3812320016 >> v1:172.21.15.33:6790/0 conn(0x7f1edc0b17d0 0x7f1edc0b2ca0 :-1 s=OPENED pgs=193050 cs=1 l=1).handle_message read tag failed 2024-03-29T11:36:46.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.889+0000 7f1edad9d640 1 --1- 172.21.15.33:0/3812320016 >> v1:172.21.15.33:6790/0 conn(0x7f1edc0b17d0 0x7f1edc0b2ca0 :-1 s=OPENED pgs=193050 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:36:46.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.889+0000 7f1ed959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.33:6790/0 2024-03-29T11:36:46.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.889+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617985829 0 0) 0x7f1edc0aae40 con 0x7f1edc142690 2024-03-29T11:36:46.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.889+0000 7f1ed959a640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.889+0000 7f1ed959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.889+0000 7f1ed959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.889+0000 7f1ed959a640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:36:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.891+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec4003520 con 0x7f1edc142690 2024-03-29T11:36:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984601459 0 0) 0x7f1ec4003520 con 0x7f1edc142690 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 >> v1:172.21.15.33:6790/0 conn(0x7f1edc0b17d0 legacy=0x7f1edc0b2ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 >> v1:172.21.15.33:6789/0 conn(0x7f1edc1422b0 legacy=0x7f1edc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 10 monclient: _finish_hunting 0 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 1 monclient: found mon.noname-c 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc151940 con 0x7f1edc142690 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 10 monclient: _finish_auth 0 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 10 monclient: _check_auth_tickets 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1ed0001690 con 0x7f1edc142690 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ed959a640 10 monclient: dump: 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ee093e640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:36:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.892+0000 7f1ee093e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.894+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ed00017f0 con 0x7f1edc142690 2024-03-29T11:36:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ed959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ed959a640 1 -- 172.21.15.33:0/3812320016 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1ed0001bc0 con 0x7f1edc142690 2024-03-29T11:36:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ed959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ed959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ed959a640 10 monclient: dump: 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:46.894 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ee093e640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ee093e640 4 set_mon_vals no callback set 2024-03-29T11:36:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ee093e640 10 monclient: shutdown 2024-03-29T11:36:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ee093e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.895+0000 7f1ee093e640 1 -- 172.21.15.33:0/3812320016 >> v1:172.21.15.153:6789/0 conn(0x7f1edc142690 legacy=0x7f1edc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.896+0000 7f1ee093e640 1 -- 172.21.15.33:0/3812320016 shutdown_connections 2024-03-29T11:36:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.896+0000 7f1ee093e640 1 -- 172.21.15.33:0/3812320016 >> 172.21.15.33:0/3812320016 conn(0x7f1edc066a00 msgr2=0x7f1edc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:46.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.899+0000 7f1ee093e640 1 -- 172.21.15.33:0/3812320016 shutdown_connections 2024-03-29T11:36:46.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.899+0000 7f1ee093e640 1 -- 172.21.15.33:0/3812320016 wait complete. 2024-03-29T11:36:46.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 10 monclient: build_initial_monmap 2024-03-29T11:36:46.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 10 monclient: monmap: 2024-03-29T11:36:46.899 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:46.899 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:46.899 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:46.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 1 librados: starting msgr at 2024-03-29T11:36:46.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 1 librados: starting objecter 2024-03-29T11:36:46.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 1 Processor -- start 2024-03-29T11:36:46.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 1 -- start start 2024-03-29T11:36:46.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 1 librados: setting wanted keys 2024-03-29T11:36:46.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 1 librados: calling monclient init 2024-03-29T11:36:46.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.901+0000 7f1ee093e640 10 monclient: init 2024-03-29T11:36:46.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 10 monclient(hunting): picked mon.noname-b con 0x7f1edc0b17d0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 10 monclient(hunting): picked mon.noname-c con 0x7f1edc1422b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 10 monclient(hunting): picked mon.noname-a con 0x7f1edc142690 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc13c9d0 con 0x7f1edc142690 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc06f9c0 con 0x7f1edc0b17d0 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc06fca0 con 0x7f1edc1422b0 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:46.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.902+0000 7f1ee093e640 10 monclient(hunting): authenticate will time out at 963.166411s 2024-03-29T11:36:46.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.908+0000 7f1eda59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1edc0b17d0 0x7f1edc06de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:59571/0 (socket says 172.21.15.33:59571) 2024-03-29T11:36:46.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.908+0000 7f1eda59c640 1 -- 172.21.15.33:0/1985064775 learned_addr learned my addr 172.21.15.33:0/1985064775 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:46.908 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 10 client.?.objecter ms_handle_connect 0x7f1edc142690 2024-03-29T11:36:46.908 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.908 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009930171 0 0) 0x7f1edc13c9d0 con 0x7f1edc142690 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.909+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc004eb0 con 0x7f1edc142690 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.910+0000 7f1ecaffd640 10 client.?.objecter ms_handle_connect 0x7f1edc1422b0 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.910+0000 7f1ecaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.910+0000 7f1ecaffd640 10 client.?.objecter ms_handle_connect 0x7f1edc0b17d0 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.910+0000 7f1ecaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698766081 0 0) 0x7f1edc06f9c0 con 0x7f1edc0b17d0 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc13c9d0 con 0x7f1edc0b17d0 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917627645 0 0) 0x7f1edc06fca0 con 0x7f1edc1422b0 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.911+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc06f9c0 con 0x7f1edc1422b0 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.912+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160441932 0 0) 0x7f1edc13c9d0 con 0x7f1edc0b17d0 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.912+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1edc06fca0 con 0x7f1edc0b17d0 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.912+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1ecc004000 con 0x7f1edc0b17d0 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.912+0000 7f1ecaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.912+0000 7f1ecaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:46.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.912+0000 7f1ecaffd640 10 monclient(hunting): dump: 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:46.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.912+0000 7f1ecaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.913+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143091564 0 0) 0x7f1ebc004eb0 con 0x7f1edc142690 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.913+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1edc13c9d0 con 0x7f1edc142690 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.913+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1ed4001c10 con 0x7f1edc142690 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.913+0000 7f1ecaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.913+0000 7f1ecaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.913+0000 7f1ecaffd640 10 monclient(hunting): dump: 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:46.916 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 961481277 0 0) 0x7f1edc13c9d0 con 0x7f1edc142690 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 >> v1:172.21.15.33:6790/0 conn(0x7f1edc1422b0 legacy=0x7f1edc13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 >> v1:172.21.15.153:6789/0 conn(0x7f1edc0b17d0 legacy=0x7f1edc06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 1 monclient: found mon.a 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc06e5a0 con 0x7f1edc142690 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ed4003970 con 0x7f1edc142690 2024-03-29T11:36:46.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ed8d99640 4 set_mon_vals no callback set 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1ed4001c10 con 0x7f1edc142690 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.914+0000 7f1ecaffd640 10 monclient: dump: 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:46.918 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 10 monclient: _renew_subs 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc071a00 con 0x7f1edc142690 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 10 monclient: _renew_subs 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc0716c0 con 0x7f1edc142690 2024-03-29T11:36:46.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.915+0000 7f1ee093e640 1 librados: init done 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.918+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f1ed4003ca0 con 0x7f1edc142690 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.918+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f1ed4084700 con 0x7f1edc142690 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.918+0000 7f1ecaffd640 10 client.4311.objecter ms_dispatch 0x7f1edc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.918+0000 7f1ecaffd640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.918+0000 7f1ecaffd640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.919+0000 7f1ecaffd640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.919+0000 7f1ee093e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.919+0000 7f1ee093e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.919+0000 7f1ee093e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:46.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.919+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc075f10 con 0x7f1edc142690 2024-03-29T11:36:46.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.919+0000 7f1eda59c640 10 monclient: get_auth_request con 0x7f1ebc06e3b0 auth_method 0 2024-03-29T11:36:46.922 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.922+0000 7f1ecaffd640 10 client.4311.objecter ms_handle_connect 0x7f1ebc06e3b0 2024-03-29T11:36:46.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.934+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ed4050120 con 0x7f1edc142690 2024-03-29T11:36:46.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.934+0000 7f1ecaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:46.934+0000 7f1ecaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:47.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.265+0000 7f1ee093e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:47.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.265+0000 7f1ee093e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:47.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.265+0000 7f1ee093e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:47.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.265+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1edc0b2270 con 0x7f1edc142690 2024-03-29T11:36:47.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.266+0000 7f1ecaffd640 1 -- 172.21.15.33:0/1985064775 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f1ed4050120 con 0x7f1edc142690 2024-03-29T11:36:47.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.266+0000 7f1ecaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:47.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.266+0000 7f1ecaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:47.265 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:47.265 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:47.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.269+0000 7f1ee093e640 10 librados: watch_flush enter 2024-03-29T11:36:47.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.270+0000 7f1ee093e640 10 librados: watch_flush exit 2024-03-29T11:36:47.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.270+0000 7f1ee093e640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T11:36:47.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.270+0000 7f1ee093e640 10 client.4311.objecter successfully canceled tick 2024-03-29T11:36:47.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.270+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 >> v1:172.21.15.153:6816/2737028074 conn(0x7f1ebc06e3b0 legacy=0x7f1ebc070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.270+0000 7f1ee093e640 10 monclient: shutdown 2024-03-29T11:36:47.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.270+0000 7f1ee093e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:47.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.270+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 >> v1:172.21.15.33:6789/0 conn(0x7f1edc142690 legacy=0x7f1edc1e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:47.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.271+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 shutdown_connections 2024-03-29T11:36:47.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.271+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 >> 172.21.15.33:0/1985064775 conn(0x7f1edc066a00 msgr2=0x7f1edc13d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:47.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.275+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 shutdown_connections 2024-03-29T11:36:47.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.276+0000 7f1ee093e640 1 -- 172.21.15.33:0/1985064775 wait complete. 2024-03-29T11:36:47.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:47.276+0000 7f1ee093e640 1 librados: shutdown 2024-03-29T11:36:47.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:54.099 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:54.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.253+0000 7ffa5e4ff640 10 monclient: get_monmap_and_config 2024-03-29T11:36:54.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.253+0000 7ffa5e4ff640 10 monclient: build_initial_monmap 2024-03-29T11:36:54.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.253+0000 7ffa5e4ff640 10 monclient: monmap: 2024-03-29T11:36:54.252 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:54.252 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:36:54.253872+0000 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:36:54.253872+0000 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:36:54.253 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:54.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.254+0000 7ffa5e4ff640 1 Processor -- start 2024-03-29T11:36:54.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.254+0000 7ffa5e4ff640 1 -- start start 2024-03-29T11:36:54.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.254+0000 7ffa5e4ff640 10 monclient: init 2024-03-29T11:36:54.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:54.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:54.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 10 monclient(hunting): picked mon.noname-b con 0x7ffa581457f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:54.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 10 monclient(hunting): picked mon.noname-c con 0x7ffa5813f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa57fff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7ffa581457f0 0x7ffa5813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:44262/0 (socket says 172.21.15.33:44262) 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 10 monclient(hunting): picked mon.noname-a con 0x7ffa580a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa580aaef0 con 0x7ffa580a9d70 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa57fff640 1 -- 172.21.15.33:0/3774831776 learned_addr learned my addr 172.21.15.33:0/3774831776 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/3774831776 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa58145bd0 con 0x7ffa5813f650 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/3774831776 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa58145eb0 con 0x7ffa581457f0 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.255+0000 7ffa5e4ff640 10 monclient(hunting): authenticate will time out at 970.519435s 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202883759 0 0) 0x7ffa58145eb0 con 0x7ffa581457f0 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa40003520 con 0x7ffa581457f0 2024-03-29T11:36:54.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280946372 0 0) 0x7ffa58145bd0 con 0x7ffa5813f650 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58145eb0 con 0x7ffa5813f650 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958036974 0 0) 0x7ffa580aaef0 con 0x7ffa580a9d70 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.256+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58145bd0 con 0x7ffa580a9d70 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275904490 0 0) 0x7ffa58145eb0 con 0x7ffa5813f650 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 >> v1:172.21.15.33:6790/0 conn(0x7ffa581457f0 legacy=0x7ffa5813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 >> v1:172.21.15.33:6789/0 conn(0x7ffa580a9d70 legacy=0x7ffa5814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 1 monclient: found mon.noname-c 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa58146340 con 0x7ffa5813f650 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ffa4c001670 con 0x7ffa5813f650 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa5e4ff640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa5e4ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa56ffd640 10 monclient: dump: 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:54.261 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.257+0000 7ffa5e4ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.258+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa4c001890 con 0x7ffa5813f650 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.258+0000 7ffa56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.258+0000 7ffa5e4ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.258+0000 7ffa5e4ff640 4 set_mon_vals no callback set 2024-03-29T11:36:54.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.259+0000 7ffa56ffd640 1 -- 172.21.15.33:0/3774831776 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ffa4c001c20 con 0x7ffa5813f650 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.259+0000 7ffa56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.259+0000 7ffa56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.259+0000 7ffa56ffd640 10 monclient: dump: 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:54.263 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:54.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.260+0000 7ffa5e4ff640 10 monclient: shutdown 2024-03-29T11:36:54.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.260+0000 7ffa5e4ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:54.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.260+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/3774831776 >> v1:172.21.15.153:6789/0 conn(0x7ffa5813f650 legacy=0x7ffa580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.260+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/3774831776 shutdown_connections 2024-03-29T11:36:54.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.260+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/3774831776 >> 172.21.15.33:0/3774831776 conn(0x7ffa58066a00 msgr2=0x7ffa58066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:54.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.260+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/3774831776 shutdown_connections 2024-03-29T11:36:54.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.260+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/3774831776 wait complete. 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.262+0000 7ffa5e4ff640 10 monclient: build_initial_monmap 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.262+0000 7ffa5e4ff640 10 monclient: monmap: 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.263+0000 7ffa5e4ff640 1 librados: starting msgr at 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.263+0000 7ffa5e4ff640 1 librados: starting objecter 2024-03-29T11:36:54.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.263+0000 7ffa5e4ff640 1 Processor -- start 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.263+0000 7ffa5e4ff640 1 -- start start 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.263+0000 7ffa5e4ff640 1 librados: setting wanted keys 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.263+0000 7ffa5e4ff640 1 librados: calling monclient init 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.263+0000 7ffa5e4ff640 10 monclient: init 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 10 monclient(hunting): picked mon.noname-c con 0x7ffa580a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 10 monclient(hunting): picked mon.noname-a con 0x7ffa5813f650 addr v1:172.21.15.33:6789/0 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 10 monclient(hunting): picked mon.noname-b con 0x7ffa581457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa581df690 con 0x7ffa5813f650 2024-03-29T11:36:54.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa580b0f80 con 0x7ffa581457f0 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa58070ef0 con 0x7ffa580a9d70 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa5e4ff640 10 monclient(hunting): authenticate will time out at 970.528304s 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa577fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7ffa5813f650 0x7ffa5806eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:39562/0 (socket says 172.21.15.33:39562) 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa577fe640 1 -- 172.21.15.33:0/5260624 learned_addr learned my addr 172.21.15.33:0/5260624 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa54ff9640 10 client.?.objecter ms_handle_connect 0x7ffa5813f650 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.264+0000 7ffa54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 client.?.objecter ms_handle_connect 0x7ffa580a9d70 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 client.?.objecter ms_handle_connect 0x7ffa581457f0 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624360942 0 0) 0x7ffa581df690 con 0x7ffa5813f650 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa44004eb0 con 0x7ffa5813f650 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323951315 0 0) 0x7ffa58070ef0 con 0x7ffa580a9d70 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.265+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa581df690 con 0x7ffa580a9d70 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.266+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903313668 0 0) 0x7ffa44004eb0 con 0x7ffa5813f650 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.266+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa58070ef0 con 0x7ffa5813f650 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.266+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040115245 0 0) 0x7ffa580b0f80 con 0x7ffa581457f0 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.266+0000 7ffa54ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.266+0000 7ffa54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.266+0000 7ffa54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:36:54.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa44004eb0 con 0x7ffa581457f0 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2353913887 0 0) 0x7ffa581df690 con 0x7ffa580a9d70 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa580b0f80 con 0x7ffa580a9d70 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ffa4c001c60 con 0x7ffa5813f650 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:54.277 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ffa480016f0 con 0x7ffa580a9d70 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.278 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 821080549 0 0) 0x7ffa58070ef0 con 0x7ffa5813f650 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 >> v1:172.21.15.33:6790/0 conn(0x7ffa580a9d70 legacy=0x7ffa581dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 >> v1:172.21.15.153:6789/0 conn(0x7ffa581457f0 legacy=0x7ffa581e61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 monclient: found mon.a 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:54.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.267+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa580711d0 con 0x7ffa5813f650 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa54ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa54ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 10 monclient: _renew_subs 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa581dfb20 con 0x7ffa5813f650 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:54.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 10 monclient: _renew_subs 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa58071b40 con 0x7ffa5813f650 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa5e4ff640 1 librados: init done 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa4c0039c0 con 0x7ffa5813f650 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.268+0000 7ffa567fc640 4 set_mon_vals no callback set 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.269+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ffa4c003d10 con 0x7ffa5813f650 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.269+0000 7ffa54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.269+0000 7ffa54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.269+0000 7ffa54ff9640 10 monclient: dump: 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:36:54.281 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.270+0000 7ffa5e4ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.270+0000 7ffa5e4ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.270+0000 7ffa5e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.270+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa38005080 con 0x7ffa5813f650 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.270+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7ffa4c003ef0 con 0x7ffa5813f650 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.271+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7ffa4c084b70 con 0x7ffa5813f650 2024-03-29T11:36:54.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.271+0000 7ffa54ff9640 10 client.4317.objecter ms_dispatch 0x7ffa58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:54.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.271+0000 7ffa54ff9640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:54.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.271+0000 7ffa54ff9640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:54.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.271+0000 7ffa54ff9640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:36:54.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.271+0000 7ffa57fff640 10 monclient: get_auth_request con 0x7ffa4406e090 auth_method 0 2024-03-29T11:36:54.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.272+0000 7ffa54ff9640 10 client.4317.objecter ms_handle_connect 0x7ffa4406e090 2024-03-29T11:36:54.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.281+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa4c0504b0 con 0x7ffa5813f650 2024-03-29T11:36:54.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.281+0000 7ffa54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:54.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.281+0000 7ffa54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:54.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.640+0000 7ffa5e4ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:54.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.640+0000 7ffa5e4ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:54.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.640+0000 7ffa5e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:36:54.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.640+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa38005330 con 0x7ffa5813f650 2024-03-29T11:36:54.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.641+0000 7ffa54ff9640 1 -- 172.21.15.33:0/5260624 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7ffa4c084220 con 0x7ffa5813f650 2024-03-29T11:36:54.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.641+0000 7ffa54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:54.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.641+0000 7ffa54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:54.641 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:36:54.641 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:54.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 10 librados: watch_flush enter 2024-03-29T11:36:54.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 10 librados: watch_flush exit 2024-03-29T11:36:54.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:36:54.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:36:54.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 >> v1:172.21.15.153:6816/2737028074 conn(0x7ffa4406e090 legacy=0x7ffa44070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 10 monclient: shutdown 2024-03-29T11:36:54.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:54.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 >> v1:172.21.15.33:6789/0 conn(0x7ffa5813f650 legacy=0x7ffa5806eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:54.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 shutdown_connections 2024-03-29T11:36:54.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.645+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 >> 172.21.15.33:0/5260624 conn(0x7ffa58066a00 msgr2=0x7ffa58066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:54.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.646+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 shutdown_connections 2024-03-29T11:36:54.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.646+0000 7ffa5e4ff640 1 -- 172.21.15.33:0/5260624 wait complete. 2024-03-29T11:36:54.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:36:54.646+0000 7ffa5e4ff640 1 librados: shutdown 2024-03-29T11:36:54.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:01.470 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:01.623 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.624+0000 7f78a7a8d640 10 monclient: get_monmap_and_config 2024-03-29T11:37:01.623 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.624+0000 7f78a7a8d640 10 monclient: build_initial_monmap 2024-03-29T11:37:01.623 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.624+0000 7f78a7a8d640 10 monclient: monmap: 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:01.624838+0000 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:01.624838+0000 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:01.624 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:01.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.625+0000 7f78a7a8d640 1 Processor -- start 2024-03-29T11:37:01.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.625+0000 7f78a7a8d640 1 -- start start 2024-03-29T11:37:01.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.625+0000 7f78a7a8d640 10 monclient: init 2024-03-29T11:37:01.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:01.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:01.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f78a01422b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:01.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f78a0142690 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f78a00b17d0 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a0142c80 con 0x7f78a00b17d0 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00aae40 con 0x7f78a0142690 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a013fbd0 con 0x7f78a01422b0 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.626+0000 7f78a7a8d640 10 monclient(hunting): authenticate will time out at 977.890407s 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a5028640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f78a0142690 0x7f78a00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:40714/0 (socket says 172.21.15.33:40714) 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a602a640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f78a00b17d0 0x7f78a00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:34804/0 (socket says 172.21.15.33:34804) 2024-03-29T11:37:01.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a5028640 1 -- 172.21.15.33:0/3846019060 learned_addr learned my addr 172.21.15.33:0/3846019060 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:01.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087115982 0 0) 0x7f78a0142c80 con 0x7f78a00b17d0 2024-03-29T11:37:01.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a4827640 10 monclient(hunting): _negotiate 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a4827640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a4827640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a4827640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.627+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f788c003520 con 0x7f78a00b17d0 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439627713 0 0) 0x7f78a00aae40 con 0x7f78a0142690 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): _negotiate 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a0142c80 con 0x7f78a0142690 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689947813 0 0) 0x7f78a013fbd0 con 0x7f78a01422b0 2024-03-29T11:37:01.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): _negotiate 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a00aae40 con 0x7f78a01422b0 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595444614 0 0) 0x7f788c003520 con 0x7f78a00b17d0 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 >> v1:172.21.15.33:6790/0 conn(0x7f78a01422b0 legacy=0x7f78a013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 >> v1:172.21.15.153:6789/0 conn(0x7f78a0142690 legacy=0x7f78a00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient: _finish_hunting 0 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 monclient: found mon.noname-a 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0151940 con 0x7f78a00b17d0 2024-03-29T11:37:01.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient: _finish_auth 0 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient: _check_auth_tickets 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7890001690 con 0x7f78a00b17d0 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.628+0000 7f78a4827640 10 monclient: dump: 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:01.634 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a7a8d640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a7a8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78900018b0 con 0x7f78a00b17d0 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a4827640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a4827640 1 -- 172.21.15.33:0/3846019060 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7890001bc0 con 0x7f78a00b17d0 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a4827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a4827640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a4827640 10 monclient: dump: 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:01.635 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a7a8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.629+0000 7f78a7a8d640 4 set_mon_vals no callback set 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.630+0000 7f78a7a8d640 10 monclient: shutdown 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.630+0000 7f78a7a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:01.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.630+0000 7f78a7a8d640 1 -- 172.21.15.33:0/3846019060 >> v1:172.21.15.33:6789/0 conn(0x7f78a00b17d0 legacy=0x7f78a00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.630+0000 7f78a7a8d640 1 -- 172.21.15.33:0/3846019060 shutdown_connections 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.630+0000 7f78a7a8d640 1 -- 172.21.15.33:0/3846019060 >> 172.21.15.33:0/3846019060 conn(0x7f78a0066a00 msgr2=0x7f78a0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.631+0000 7f78a7a8d640 1 -- 172.21.15.33:0/3846019060 shutdown_connections 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.631+0000 7f78a7a8d640 1 -- 172.21.15.33:0/3846019060 wait complete. 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 10 monclient: build_initial_monmap 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 10 monclient: monmap: 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:01.637 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 1 librados: starting msgr at 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 1 librados: starting objecter 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 1 Processor -- start 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 1 -- start start 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 1 librados: setting wanted keys 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 1 librados: calling monclient init 2024-03-29T11:37:01.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.632+0000 7f78a7a8d640 10 monclient: init 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f78a00b17d0 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f78a01422b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f78a0142690 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00ac590 con 0x7f78a0142690 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00b0b60 con 0x7f78a00b17d0 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a0137b30 con 0x7f78a01422b0 2024-03-29T11:37:01.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a7a8d640 10 monclient(hunting): authenticate will time out at 977.897368s 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a5829640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f78a00b17d0 0x7f78a01dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:40918/0 (socket says 172.21.15.33:40918) 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.633+0000 7f78a5829640 1 -- 172.21.15.33:0/244400636 learned_addr learned my addr 172.21.15.33:0/244400636 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 client.?.objecter ms_handle_connect 0x7f78a0142690 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37041479 0 0) 0x7f78a00ac590 con 0x7f78a0142690 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7884004eb0 con 0x7f78a0142690 2024-03-29T11:37:01.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 client.?.objecter ms_handle_connect 0x7f78a01422b0 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496474679 0 0) 0x7f78a0137b30 con 0x7f78a01422b0 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.634+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a00ac590 con 0x7f78a01422b0 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3175277992 0 0) 0x7f78a00ac590 con 0x7f78a01422b0 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78a0137b30 con 0x7f78a01422b0 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f789c001910 con 0x7f78a01422b0 2024-03-29T11:37:01.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient(hunting): dump: 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:01.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878714585 0 0) 0x7f7884004eb0 con 0x7f78a0142690 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78a00ac590 con 0x7f78a0142690 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7890001550 con 0x7f78a0142690 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient(hunting): dump: 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.646 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1745515510 0 0) 0x7f78a0137b30 con 0x7f78a01422b0 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 >> v1:172.21.15.153:6789/0 conn(0x7f78a00b17d0 legacy=0x7f78a01dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 >> v1:172.21.15.33:6789/0 conn(0x7f78a0142690 legacy=0x7f78a0137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 monclient: found mon.c 2024-03-29T11:37:01.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00b0dd0 con 0x7f78a01422b0 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 10 monclient: _renew_subs 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a00b10b0 con 0x7f78a01422b0 2024-03-29T11:37:01.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 10 monclient: _renew_subs 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a00af180 con 0x7f78a01422b0 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.635+0000 7f78a7a8d640 1 librados: init done 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.636+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f789c0047e0 con 0x7f78a01422b0 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.636+0000 7f78967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.636+0000 7f7897fff640 4 set_mon_vals no callback set 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.636+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f789c004af0 con 0x7f78a01422b0 2024-03-29T11:37:01.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.636+0000 7f78967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.636+0000 7f78967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.636+0000 7f78967fc640 10 monclient: dump: 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:01.650 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.638+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f789c01aac0 con 0x7f78a01422b0 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.638+0000 7f78a7a8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.638+0000 7f78a7a8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.638+0000 7f78a7a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.638+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888005080 con 0x7f78a01422b0 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.639+0000 7f78a5829640 10 monclient: get_auth_request con 0x7f788406de30 auth_method 0 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.639+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f789c084980 con 0x7f78a01422b0 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.639+0000 7f78967fc640 10 client.4337.objecter ms_dispatch 0x7f78a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:01.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.639+0000 7f78967fc640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:01.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.639+0000 7f78967fc640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:01.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.639+0000 7f78967fc640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:37:01.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.640+0000 7f78967fc640 10 client.4337.objecter ms_handle_connect 0x7f788406de30 2024-03-29T11:37:01.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.646+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f789c050260 con 0x7f78a01422b0 2024-03-29T11:37:01.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.647+0000 7f78967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:01.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:01.647+0000 7f78967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:02.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.033+0000 7f78a7a8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:02.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.033+0000 7f78a7a8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:02.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.033+0000 7f78a7a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:02.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.033+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7888005330 con 0x7f78a01422b0 2024-03-29T11:37:02.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.035+0000 7f78967fc640 1 -- 172.21.15.33:0/244400636 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f789c052190 con 0x7f78a01422b0 2024-03-29T11:37:02.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.035+0000 7f78967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:02.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.035+0000 7f78967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:02.034 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:02.035 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:02.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 10 librados: watch_flush enter 2024-03-29T11:37:02.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 10 librados: watch_flush exit 2024-03-29T11:37:02.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:37:02.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:37:02.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 >> v1:172.21.15.153:6816/2737028074 conn(0x7f788406de30 legacy=0x7f78840702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:02.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 10 monclient: shutdown 2024-03-29T11:37:02.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:02.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 >> v1:172.21.15.33:6790/0 conn(0x7f78a01422b0 legacy=0x7f78a0136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:02.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.038+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 shutdown_connections 2024-03-29T11:37:02.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.039+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 >> 172.21.15.33:0/244400636 conn(0x7f78a0066a00 msgr2=0x7f78a013d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:02.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.039+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 shutdown_connections 2024-03-29T11:37:02.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.039+0000 7f78a7a8d640 1 -- 172.21.15.33:0/244400636 wait complete. 2024-03-29T11:37:02.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:02.039+0000 7f78a7a8d640 1 librados: shutdown 2024-03-29T11:37:02.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:08.858 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:08.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.991+0000 7ff6e111c640 10 monclient: get_monmap_and_config 2024-03-29T11:37:08.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.991+0000 7ff6e111c640 10 monclient: build_initial_monmap 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.991+0000 7ff6e111c640 10 monclient: monmap: 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:08.993163+0000 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:08.993163+0000 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:08.992 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:08.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.992+0000 7ff6e111c640 1 Processor -- start 2024-03-29T11:37:08.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.992+0000 7ff6e111c640 1 -- start start 2024-03-29T11:37:08.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.992+0000 7ff6e111c640 10 monclient: init 2024-03-29T11:37:08.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 10 monclient(hunting): picked mon.noname-a con 0x7ff6dc1457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 10 monclient(hunting): picked mon.noname-c con 0x7ff6dc13f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 10 monclient(hunting): picked mon.noname-b con 0x7ff6dc0a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc0aaef0 con 0x7ff6dc1457f0 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc145bd0 con 0x7ff6dc13f650 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc145eb0 con 0x7ff6dc0a9d70 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6e111c640 10 monclient(hunting): authenticate will time out at 985.258002s 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6db59e640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7ff6dc0a9d70 0x7ff6dc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:40568/0 (socket says 172.21.15.33:40568) 2024-03-29T11:37:08.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.993+0000 7ff6db59e640 1 -- 172.21.15.33:0/3951673070 learned_addr learned my addr 172.21.15.33:0/3951673070 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928710205 0 0) 0x7ff6dc145bd0 con 0x7ff6dc13f650 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c4003520 con 0x7ff6dc13f650 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099302088 0 0) 0x7ff6dc145eb0 con 0x7ff6dc0a9d70 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.996 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:37:08.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc145bd0 con 0x7ff6dc0a9d70 2024-03-29T11:37:08.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571510845 0 0) 0x7ff6dc145bd0 con 0x7ff6dc0a9d70 2024-03-29T11:37:08.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 >> v1:172.21.15.153:6789/0 conn(0x7ff6dc13f650 legacy=0x7ff6dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 >> v1:172.21.15.33:6789/0 conn(0x7ff6dc1457f0 legacy=0x7ff6dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 monclient: found mon.noname-b 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc146340 con 0x7ff6dc0a9d70 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ff6d4001970 con 0x7ff6dc0a9d70 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:08.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.996+0000 7ff6d9d9b640 10 monclient: dump: 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.997+0000 7ff6e111c640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.997+0000 7ff6e111c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.997+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d4001ad0 con 0x7ff6dc0a9d70 2024-03-29T11:37:08.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.997+0000 7ff6d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:09.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.997+0000 7ff6e111c640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.997+0000 7ff6e111c640 4 set_mon_vals no callback set 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6d9d9b640 1 -- 172.21.15.33:0/3951673070 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ff6d4001e60 con 0x7ff6dc0a9d70 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6d9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6d9d9b640 10 monclient: dump: 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:09.001 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6e111c640 10 monclient: shutdown 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6e111c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6e111c640 1 -- 172.21.15.33:0/3951673070 >> v1:172.21.15.33:6790/0 conn(0x7ff6dc0a9d70 legacy=0x7ff6dc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6e111c640 1 -- 172.21.15.33:0/3951673070 shutdown_connections 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6e111c640 1 -- 172.21.15.33:0/3951673070 >> 172.21.15.33:0/3951673070 conn(0x7ff6dc066a00 msgr2=0x7ff6dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6e111c640 1 -- 172.21.15.33:0/3951673070 shutdown_connections 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.998+0000 7ff6e111c640 1 -- 172.21.15.33:0/3951673070 wait complete. 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.999+0000 7ff6e111c640 10 monclient: build_initial_monmap 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:08.999+0000 7ff6e111c640 10 monclient: monmap: 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:09.002 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.000+0000 7ff6e111c640 1 librados: starting msgr at 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.000+0000 7ff6e111c640 1 librados: starting objecter 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.000+0000 7ff6e111c640 1 Processor -- start 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.000+0000 7ff6e111c640 1 -- start start 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.000+0000 7ff6e111c640 1 librados: setting wanted keys 2024-03-29T11:37:09.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.000+0000 7ff6e111c640 1 librados: calling monclient init 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.000+0000 7ff6e111c640 10 monclient: init 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 10 monclient(hunting): picked mon.noname-a con 0x7ff6dc0a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 10 monclient(hunting): picked mon.noname-c con 0x7ff6dc13f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6da59c640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7ff6dc13f650 0x7ff6dc142740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:41790/0 (socket says 172.21.15.33:41790) 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6da59c640 1 -- 172.21.15.33:0/2575954369 learned_addr learned my addr 172.21.15.33:0/2575954369 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 10 monclient(hunting): picked mon.noname-b con 0x7ff6dc1457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:09.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc140450 con 0x7ff6dc0a9d70 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc1409f0 con 0x7ff6dc1457f0 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc145110 con 0x7ff6dc13f650 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.001+0000 7ff6e111c640 10 monclient(hunting): authenticate will time out at 985.266464s 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 client.?.objecter ms_handle_connect 0x7ff6dc13f650 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 client.?.objecter ms_handle_connect 0x7ff6dc1457f0 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615366864 0 0) 0x7ff6dc145110 con 0x7ff6dc13f650 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:37:09.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c8004eb0 con 0x7ff6dc13f650 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324328460 0 0) 0x7ff6dc1409f0 con 0x7ff6dc1457f0 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.002+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc145110 con 0x7ff6dc1457f0 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264745108 0 0) 0x7ff6c8004eb0 con 0x7ff6dc13f650 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6dc1409f0 con 0x7ff6dc13f650 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ff6cc0019c0 con 0x7ff6dc13f650 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 10 monclient(hunting): dump: 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:09.011 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2543519812 0 0) 0x7ff6dc1409f0 con 0x7ff6dc13f650 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 >> v1:172.21.15.153:6789/0 conn(0x7ff6dc1457f0 legacy=0x7ff6dc1e1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.003+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 >> v1:172.21.15.33:6789/0 conn(0x7ff6dc0a9d70 legacy=0x7ff6dc142030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 1 monclient: found mon.c 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:09.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc1452a0 con 0x7ff6dc13f650 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 10 monclient: _renew_subs 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc137070 con 0x7ff6dc13f650 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 10 monclient: _renew_subs 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc13fda0 con 0x7ff6dc13f650 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.004+0000 7ff6e111c640 1 librados: init done 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6cc001ba0 con 0x7ff6dc13f650 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:09.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7ff6cc003a60 con 0x7ff6dc13f650 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6d959a640 4 set_mon_vals no callback set 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6c37fe640 10 monclient: dump: 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:09.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.005+0000 7ff6e111c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.006+0000 7ff6e111c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.006+0000 7ff6e111c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.006+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6bc005080 con 0x7ff6dc13f650 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.007+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7ff6cc003bc0 con 0x7ff6dc13f650 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.008+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7ff6cc084b90 con 0x7ff6dc13f650 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.008+0000 7ff6c37fe640 10 client.4343.objecter ms_dispatch 0x7ff6dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.008+0000 7ff6c37fe640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.008+0000 7ff6c37fe640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.008+0000 7ff6dad9d640 10 monclient: get_auth_request con 0x7ff6c806dd20 auth_method 0 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.008+0000 7ff6c37fe640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:37:09.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.008+0000 7ff6c37fe640 10 client.4343.objecter ms_handle_connect 0x7ff6c806dd20 2024-03-29T11:37:09.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.013+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6cc050470 con 0x7ff6dc13f650 2024-03-29T11:37:09.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.013+0000 7ff6c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:09.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.013+0000 7ff6c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:09.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.327+0000 7ff6e111c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:09.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.327+0000 7ff6e111c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:09.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.327+0000 7ff6e111c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:09.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.327+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6bc005330 con 0x7ff6dc13f650 2024-03-29T11:37:09.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.328+0000 7ff6c37fe640 1 -- 172.21.15.33:0/2575954369 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7ff6cc0523a0 con 0x7ff6dc13f650 2024-03-29T11:37:09.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.328+0000 7ff6c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:09.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.328+0000 7ff6c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:09.329 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:09.329 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:09.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 10 librados: watch_flush enter 2024-03-29T11:37:09.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 10 librados: watch_flush exit 2024-03-29T11:37:09.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 >> v1:172.21.15.153:6816/2737028074 conn(0x7ff6c806dd20 legacy=0x7ff6c80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 10 monclient: shutdown 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 >> v1:172.21.15.33:6790/0 conn(0x7ff6dc13f650 legacy=0x7ff6dc142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 shutdown_connections 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.332+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 >> 172.21.15.33:0/2575954369 conn(0x7ff6dc066a00 msgr2=0x7ff6dc13b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.333+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 shutdown_connections 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.333+0000 7ff6e111c640 1 -- 172.21.15.33:0/2575954369 wait complete. 2024-03-29T11:37:09.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:09.333+0000 7ff6e111c640 1 librados: shutdown 2024-03-29T11:37:09.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:16.152 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:16.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.305+0000 7f462c845640 10 monclient: get_monmap_and_config 2024-03-29T11:37:16.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.305+0000 7f462c845640 10 monclient: build_initial_monmap 2024-03-29T11:37:16.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.305+0000 7f462c845640 10 monclient: monmap: 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:16.307141+0000 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:16.307141+0000 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.306+0000 7f462c845640 1 Processor -- start 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.306+0000 7f462c845640 1 -- start start 2024-03-29T11:37:16.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.306+0000 7f462c845640 10 monclient: init 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 10 monclient(hunting): picked mon.noname-a con 0x7f46280b0770 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 10 monclient(hunting): picked mon.noname-c con 0x7f462814a700 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 10 monclient(hunting): picked mon.noname-b con 0x7f46280ac5a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280aba70 con 0x7f46280b0770 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280aecb0 con 0x7f462814a700 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280af150 con 0x7f46280ac5a0 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:16.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.307+0000 7f462c845640 10 monclient(hunting): authenticate will time out at 992.572481s 2024-03-29T11:37:16.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.310+0000 7f4626d9d640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f46280ac5a0 0x7f4628151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:37223/0 (socket says 172.21.15.33:37223) 2024-03-29T11:37:16.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.310+0000 7f4626d9d640 1 -- 172.21.15.33:0/1975970801 learned_addr learned my addr 172.21.15.33:0/1975970801 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:16.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788129240 0 0) 0x7f46280af150 con 0x7f46280ac5a0 2024-03-29T11:37:16.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:37:16.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4610003520 con 0x7f46280ac5a0 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021599021 0 0) 0x7f4610003520 con 0x7f46280ac5a0 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 >> v1:172.21.15.153:6789/0 conn(0x7f462814a700 legacy=0x7f46280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 >> v1:172.21.15.33:6789/0 conn(0x7f46280b0770 legacy=0x7f4628149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 monclient: found mon.noname-b 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280ac9d0 con 0x7f46280ac5a0 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: _finish_auth 0 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f46200019f0 con 0x7f46280ac5a0 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: dump: 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4620001c50 con 0x7f46280ac5a0 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 1 -- 172.21.15.33:0/1975970801 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f46200019f0 con 0x7f46280ac5a0 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462559a640 10 monclient: dump: 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:16.314 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462c845640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462c845640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.311+0000 7f462c845640 4 set_mon_vals no callback set 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.312+0000 7f462c845640 10 monclient: shutdown 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.312+0000 7f462c845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.312+0000 7f462c845640 1 -- 172.21.15.33:0/1975970801 >> v1:172.21.15.33:6790/0 conn(0x7f46280ac5a0 legacy=0x7f4628151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.313+0000 7f462c845640 1 -- 172.21.15.33:0/1975970801 shutdown_connections 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.313+0000 7f462c845640 1 -- 172.21.15.33:0/1975970801 >> 172.21.15.33:0/1975970801 conn(0x7f462806b660 msgr2=0x7f462806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.313+0000 7f462c845640 1 -- 172.21.15.33:0/1975970801 shutdown_connections 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.313+0000 7f462c845640 1 -- 172.21.15.33:0/1975970801 wait complete. 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.314+0000 7f462c845640 10 monclient: build_initial_monmap 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.314+0000 7f462c845640 10 monclient: monmap: 2024-03-29T11:37:16.316 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.314+0000 7f462c845640 1 librados: starting msgr at 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.314+0000 7f462c845640 1 librados: starting objecter 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 1 Processor -- start 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 1 -- start start 2024-03-29T11:37:16.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 1 librados: setting wanted keys 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 1 librados: calling monclient init 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient: init 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient(hunting): picked mon.noname-b con 0x7f46280ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient(hunting): picked mon.noname-c con 0x7f46280b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient(hunting): picked mon.noname-a con 0x7f462814a700 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:16.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4628122b70 con 0x7f462814a700 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4628121040 con 0x7f46280ac5a0 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462811fec0 con 0x7f46280b0770 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.315+0000 7f462c845640 10 monclient(hunting): authenticate will time out at 992.580638s 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.317+0000 7f462659c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f46280ac5a0 0x7f46281e3d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:52100/0 (socket says 172.21.15.33:52100) 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.317+0000 7f462659c640 1 -- 172.21.15.33:0/3013999855 learned_addr learned my addr 172.21.15.33:0/3013999855 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 client.?.objecter ms_handle_connect 0x7f46280ac5a0 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122794239 0 0) 0x7f4628121040 con 0x7f46280ac5a0 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:37:16.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4608004eb0 con 0x7f46280ac5a0 2024-03-29T11:37:16.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 208219706 0 0) 0x7f4608004eb0 con 0x7f46280ac5a0 2024-03-29T11:37:16.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4628121040 con 0x7f46280ac5a0 2024-03-29T11:37:16.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f461c001960 con 0x7f46280ac5a0 2024-03-29T11:37:16.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:16.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): dump: 2024-03-29T11:37:16.322 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.318+0000 7f4616ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 client.?.objecter ms_handle_connect 0x7f46280b0770 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977966004 0 0) 0x7f462811fec0 con 0x7f46280b0770 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:16.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4608004eb0 con 0x7f46280b0770 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097821657 0 0) 0x7f4608004eb0 con 0x7f46280b0770 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f462811fec0 con 0x7f46280b0770 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f46180016f0 con 0x7f46280b0770 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.319+0000 7f4616ffd640 10 monclient(hunting): dump: 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:16.324 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3255147599 0 0) 0x7f462811fec0 con 0x7f46280b0770 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 >> v1:172.21.15.153:6789/0 conn(0x7f46280ac5a0 legacy=0x7f46281e3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 >> v1:172.21.15.33:6789/0 conn(0x7f462814a700 legacy=0x7f4628147fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 1 monclient: found mon.c 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46281212b0 con 0x7f46280b0770 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f4616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:16.325 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:37:16.326 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 10 monclient: _renew_subs 2024-03-29T11:37:16.326 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:16.326 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46281486e0 con 0x7f46280b0770 2024-03-29T11:37:16.326 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:16.326 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 10 monclient: _renew_subs 2024-03-29T11:37:16.326 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:16.326 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4628123270 con 0x7f46280b0770 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.320+0000 7f462c845640 1 librados: init done 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.321+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4618001be0 con 0x7f46280b0770 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.321+0000 7f4616ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.321+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4618003790 con 0x7f46280b0770 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.321+0000 7f4616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.321+0000 7f4616ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.321+0000 7f4616ffd640 10 monclient: dump: 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:16.327 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.321+0000 7f4617fff640 4 set_mon_vals no callback set 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.322+0000 7f462c845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.322+0000 7f462c845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.322+0000 7f462c845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.322+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c005080 con 0x7f46280b0770 2024-03-29T11:37:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.323+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f4618019760 con 0x7f46280b0770 2024-03-29T11:37:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.323+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f4618083690 con 0x7f46280b0770 2024-03-29T11:37:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.323+0000 7f4616ffd640 10 client.4349.objecter ms_dispatch 0x7f4628000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.323+0000 7f4616ffd640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.323+0000 7f4616ffd640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.323+0000 7f4616ffd640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:37:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.324+0000 7f462659c640 10 monclient: get_auth_request con 0x7f460806df80 auth_method 0 2024-03-29T11:37:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.324+0000 7f4616ffd640 10 client.4349.objecter ms_handle_connect 0x7f460806df80 2024-03-29T11:37:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.330+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f461804ef70 con 0x7f46280b0770 2024-03-29T11:37:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.330+0000 7f4616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.330+0000 7f4616ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:16.656 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.656+0000 7f462c845640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.656 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.656+0000 7f462c845640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.656 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.656+0000 7f462c845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:16.656 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.656+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f460c005330 con 0x7f46280b0770 2024-03-29T11:37:16.657 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.657+0000 7f4616ffd640 1 -- 172.21.15.33:0/3013999855 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f4618050ea0 con 0x7f46280b0770 2024-03-29T11:37:16.657 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.657+0000 7f4616ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.658 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.657+0000 7f4616ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:16.658 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:16.658 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:16.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.661+0000 7f462c845640 10 librados: watch_flush enter 2024-03-29T11:37:16.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.661+0000 7f462c845640 10 librados: watch_flush exit 2024-03-29T11:37:16.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.661+0000 7f462c845640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:37:16.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.661+0000 7f462c845640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.661+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 >> v1:172.21.15.153:6816/2737028074 conn(0x7f460806df80 legacy=0x7f4608070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.661+0000 7f462c845640 10 monclient: shutdown 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.661+0000 7f462c845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.662+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 >> v1:172.21.15.33:6790/0 conn(0x7f46280b0770 legacy=0x7f4628120930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.662+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 shutdown_connections 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.662+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 >> 172.21.15.33:0/3013999855 conn(0x7f462806b660 msgr2=0x7f46280b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.662+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 shutdown_connections 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.662+0000 7f462c845640 1 -- 172.21.15.33:0/3013999855 wait complete. 2024-03-29T11:37:16.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:16.662+0000 7f462c845640 1 librados: shutdown 2024-03-29T11:37:16.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:23.486 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:23.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.635+0000 7f0967fff640 10 monclient: get_monmap_and_config 2024-03-29T11:37:23.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.635+0000 7f0967fff640 10 monclient: build_initial_monmap 2024-03-29T11:37:23.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.635+0000 7f0967fff640 10 monclient: monmap: 2024-03-29T11:37:23.636 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:23.637323+0000 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:23.637323+0000 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:23.637 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:23.639 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.639+0000 7f0967fff640 1 Processor -- start 2024-03-29T11:37:23.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.639+0000 7f0967fff640 1 -- start start 2024-03-29T11:37:23.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.639+0000 7f0967fff640 10 monclient: init 2024-03-29T11:37:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.640+0000 7f0967fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.640+0000 7f0967fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.643+0000 7f0967fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0960006a70 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.643+0000 7f0967fff640 10 monclient(hunting): picked mon.noname-b con 0x7f096002b100 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.643+0000 7f0967fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0960029c30 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.643+0000 7f0967fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096002a270 con 0x7f0960029c30 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.644+0000 7f0967fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096002c440 con 0x7f0960006a70 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.644+0000 7f0967fff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09600c9440 con 0x7f096002b100 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.644+0000 7f0967fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.644+0000 7f0967fff640 10 monclient(hunting): authenticate will time out at 999.908945s 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.644+0000 7f0966ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0960006a70 0x7f09600c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:41169/0 (socket says 172.21.15.33:41169) 2024-03-29T11:37:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.644+0000 7f0966ffd640 1 -- 172.21.15.33:0/85453302 learned_addr learned my addr 172.21.15.33:0/85453302 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.645+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884931720 0 0) 0x7f096002c440 con 0x7f0960006a70 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.645+0000 7f0965ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.645+0000 7f0965ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.645+0000 7f0965ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.645+0000 7f0965ffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.645+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0950003520 con 0x7f0960006a70 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.646+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309165749 0 0) 0x7f0950003520 con 0x7f0960006a70 2024-03-29T11:37:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.646+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 >> v1:172.21.15.33:6790/0 conn(0x7f096002b100 legacy=0x7f09600cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 >> v1:172.21.15.33:6789/0 conn(0x7f0960029c30 legacy=0x7f09600d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 1 monclient: found mon.noname-c 2024-03-29T11:37:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:37:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600c95d0 con 0x7f0960006a70 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0967fff640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0967fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f096804f730 con 0x7f0960006a70 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: dump: 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.652 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09680008d0 con 0x7f0960006a70 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 1 -- 172.21.15.33:0/85453302 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f09680758b0 con 0x7f0960006a70 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0965ffb640 10 monclient: dump: 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.653 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0967fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.648+0000 7f0967fff640 4 set_mon_vals no callback set 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.650+0000 7f0967fff640 10 monclient: shutdown 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.650+0000 7f0967fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:23.654 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.650+0000 7f0967fff640 1 -- 172.21.15.33:0/85453302 >> v1:172.21.15.153:6789/0 conn(0x7f0960006a70 legacy=0x7f09600c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.656 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.655+0000 7f0967fff640 1 -- 172.21.15.33:0/85453302 shutdown_connections 2024-03-29T11:37:23.656 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.655+0000 7f0967fff640 1 -- 172.21.15.33:0/85453302 >> 172.21.15.33:0/85453302 conn(0x7f09600058f0 msgr2=0x7f0960005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:23.656 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.655+0000 7f0967fff640 1 -- 172.21.15.33:0/85453302 shutdown_connections 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.656+0000 7f0967fff640 1 -- 172.21.15.33:0/85453302 wait complete. 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.656+0000 7f0967fff640 10 monclient: build_initial_monmap 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.656+0000 7f0967fff640 10 monclient: monmap: 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:23.658 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.657+0000 7f0967fff640 1 librados: starting msgr at 2024-03-29T11:37:23.659 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.657+0000 7f0967fff640 1 librados: starting objecter 2024-03-29T11:37:23.659 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.657+0000 7f0967fff640 1 Processor -- start 2024-03-29T11:37:23.660 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.659+0000 7f0967fff640 1 -- start start 2024-03-29T11:37:23.660 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.659+0000 7f0967fff640 1 librados: setting wanted keys 2024-03-29T11:37:23.660 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.659+0000 7f0967fff640 1 librados: calling monclient init 2024-03-29T11:37:23.660 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.659+0000 7f0967fff640 10 monclient: init 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0960006a70 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0960029c30 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0960034d30 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09600cb890 con 0x7f0960029c30 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09600cc890 con 0x7f0960006a70 2024-03-29T11:37:23.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09600cbb70 con 0x7f0960034d30 2024-03-29T11:37:23.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:23.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.660+0000 7f0967fff640 10 monclient(hunting): authenticate will time out at 999.925543s 2024-03-29T11:37:23.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.663+0000 7f0966ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0960006a70 0x7f09600c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:41277/0 (socket says 172.21.15.33:41277) 2024-03-29T11:37:23.663 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.663+0000 7f0966ffd640 1 -- 172.21.15.33:0/2222074691 learned_addr learned my addr 172.21.15.33:0/2222074691 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 client.?.objecter ms_handle_connect 0x7f0960006a70 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172708781 0 0) 0x7f09600cc890 con 0x7f0960006a70 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:37:23.664 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0954004eb0 con 0x7f0960006a70 2024-03-29T11:37:23.665 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 682728815 0 0) 0x7f0954004eb0 con 0x7f0960006a70 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09600cc890 con 0x7f0960006a70 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f096804f400 con 0x7f0960006a70 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.666 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.664+0000 7f094f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.665+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3145165073 0 0) 0x7f09600cc890 con 0x7f0960006a70 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.665+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 >> v1:172.21.15.33:6790/0 conn(0x7f0960034d30 legacy=0x7f0960174bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:23.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.665+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 >> v1:172.21.15.33:6789/0 conn(0x7f0960029c30 legacy=0x7f09600c9980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:23.670 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.668+0000 7f094f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:23.670 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.668+0000 7f094f7fe640 1 monclient: found mon.b 2024-03-29T11:37:23.670 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.668+0000 7f094f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:23.670 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.668+0000 7f094f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:23.670 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.668+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600cc370 con 0x7f0960006a70 2024-03-29T11:37:23.670 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:23.670 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f0967fff640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f0967fff640 10 monclient: _renew_subs 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f0967fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f0967fff640 1 -- 172.21.15.33:0/2222074691 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09600cbf90 con 0x7f0960006a70 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f096804f730 con 0x7f0960006a70 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0968067d90 con 0x7f0960006a70 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:23.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f094f7fe640 10 monclient: dump: 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:23.672 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.669+0000 7f09657fa640 4 set_mon_vals no callback set 2024-03-29T11:37:23.673 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.671+0000 7f0967fff640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.671+0000 7f0967fff640 10 monclient: _renew_subs 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.671+0000 7f0967fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.671+0000 7f0967fff640 1 -- 172.21.15.33:0/2222074691 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09600c3010 con 0x7f0960006a70 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.671+0000 7f0967fff640 1 librados: init done 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.672+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f0968067ef0 con 0x7f0960006a70 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.672+0000 7f094d7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.672+0000 7f094d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.672+0000 7f094d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.672+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0960004cd0 con 0x7f0960006a70 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.672+0000 7f09667fc640 10 monclient: get_auth_request con 0x7f095406da30 auth_method 0 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.673+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f09681009b0 con 0x7f0960006a70 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.673+0000 7f094f7fe640 10 client.4327.objecter ms_dispatch 0x7f0968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.673+0000 7f094f7fe640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.673+0000 7f094f7fe640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:23.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.673+0000 7f094f7fe640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T11:37:23.675 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.673+0000 7f094f7fe640 10 client.4327.objecter ms_handle_connect 0x7f095406da30 2024-03-29T11:37:23.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.680+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09600c3010 con 0x7f0960006a70 2024-03-29T11:37:23.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.680+0000 7f094f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:23.680+0000 7f094f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:24.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.005+0000 7f094d7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:24.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.005+0000 7f094d7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:24.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.005+0000 7f094d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:24.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.005+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09600c3010 con 0x7f0960006a70 2024-03-29T11:37:24.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.010+0000 7f094f7fe640 1 -- 172.21.15.33:0/2222074691 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f09600c3010 con 0x7f0960006a70 2024-03-29T11:37:24.014 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:24.014 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.010+0000 7f094f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.010+0000 7f094f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.014+0000 7f094d7fa640 10 librados: watch_flush enter 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.014+0000 7f094d7fa640 10 librados: watch_flush exit 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.014+0000 7f094d7fa640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.014+0000 7f094d7fa640 10 client.4327.objecter successfully canceled tick 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.014+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 >> v1:172.21.15.153:6816/2737028074 conn(0x7f095406da30 legacy=0x7f095406fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.014+0000 7f094d7fa640 10 monclient: shutdown 2024-03-29T11:37:24.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.014+0000 7f094d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:24.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.015+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 >> v1:172.21.15.153:6789/0 conn(0x7f0960006a70 legacy=0x7f09600c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:24.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.015+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 shutdown_connections 2024-03-29T11:37:24.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.015+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 >> 172.21.15.33:0/2222074691 conn(0x7f09600058f0 msgr2=0x7f09600c3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:24.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.018+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 shutdown_connections 2024-03-29T11:37:24.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.018+0000 7f094d7fa640 1 -- 172.21.15.33:0/2222074691 wait complete. 2024-03-29T11:37:24.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:24.018+0000 7f094d7fa640 1 librados: shutdown 2024-03-29T11:37:24.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:30.843 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:31.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.000+0000 7f8d7dc58640 10 monclient: get_monmap_and_config 2024-03-29T11:37:31.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.001+0000 7f8d7dc58640 10 monclient: build_initial_monmap 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.001+0000 7f8d7dc58640 10 monclient: monmap: 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:31.002635+0000 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:31.002635+0000 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:31.002 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:31.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.003+0000 7f8d7dc58640 1 Processor -- start 2024-03-29T11:37:31.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.003+0000 7f8d7dc58640 1 -- start start 2024-03-29T11:37:31.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.003+0000 7f8d7dc58640 10 monclient: init 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 10 monclient(hunting): picked mon.noname-a con 0x7f8d700389e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 10 monclient(hunting): picked mon.noname-b con 0x7f8d70038dc0 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 10 monclient(hunting): picked mon.noname-c con 0x7f8d70030660 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700c54c0 con 0x7f8d700389e0 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70036430 con 0x7f8d70030660 2024-03-29T11:37:31.004 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70039840 con 0x7f8d70038dc0 2024-03-29T11:37:31.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:31.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.004+0000 7f8d7dc58640 10 monclient(hunting): authenticate will time out at 1007.268800s 2024-03-29T11:37:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.007+0000 7f8d7cc56640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f8d700389e0 0x7f8d70033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:55381/0 (socket says 172.21.15.33:55381) 2024-03-29T11:37:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.007+0000 7f8d7cc56640 1 -- 172.21.15.33:0/2186249699 learned_addr learned my addr 172.21.15.33:0/2186249699 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953599013 0 0) 0x7f8d70036430 con 0x7f8d70030660 2024-03-29T11:37:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:37:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d68003520 con 0x7f8d70030660 2024-03-29T11:37:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898616349 0 0) 0x7f8d700c54c0 con 0x7f8d700389e0 2024-03-29T11:37:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:37:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d70036430 con 0x7f8d700389e0 2024-03-29T11:37:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.008+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 567661920 0 0) 0x7f8d70036430 con 0x7f8d700389e0 2024-03-29T11:37:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.009+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 >> v1:172.21.15.33:6790/0 conn(0x7f8d70038dc0 legacy=0x7f8d700cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.010+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 >> v1:172.21.15.153:6789/0 conn(0x7f8d70030660 legacy=0x7f8d700d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.010+0000 7f8d777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.010+0000 7f8d777fe640 1 monclient: found mon.noname-a 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.010+0000 7f8d777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70035690 con 0x7f8d700389e0 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d7dc58640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d7dc58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8d6c001690 con 0x7f8d700389e0 2024-03-29T11:37:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 10 monclient: dump: 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d7dc58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:31.014 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d6c001920 con 0x7f8d700389e0 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d7dc58640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.011+0000 7f8d7dc58640 4 set_mon_vals no callback set 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.012+0000 7f8d777fe640 1 -- 172.21.15.33:0/2186249699 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8d6c001cb0 con 0x7f8d700389e0 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.012+0000 7f8d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.012+0000 7f8d777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.012+0000 7f8d777fe640 10 monclient: dump: 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:31.015 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:31.016 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:31.016 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:31.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.012+0000 7f8d7dc58640 10 monclient: shutdown 2024-03-29T11:37:31.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.012+0000 7f8d7dc58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:31.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.012+0000 7f8d7dc58640 1 -- 172.21.15.33:0/2186249699 >> v1:172.21.15.33:6789/0 conn(0x7f8d700389e0 legacy=0x7f8d70033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.016 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.013+0000 7f8d7dc58640 1 -- 172.21.15.33:0/2186249699 shutdown_connections 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.013+0000 7f8d7dc58640 1 -- 172.21.15.33:0/2186249699 >> 172.21.15.33:0/2186249699 conn(0x7f8d70005c00 msgr2=0x7f8d70006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.013+0000 7f8d7dc58640 1 -- 172.21.15.33:0/2186249699 shutdown_connections 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.013+0000 7f8d7dc58640 1 -- 172.21.15.33:0/2186249699 wait complete. 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.014+0000 7f8d7dc58640 10 monclient: build_initial_monmap 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.014+0000 7f8d7dc58640 10 monclient: monmap: 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:31.017 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.014+0000 7f8d7dc58640 1 librados: starting msgr at 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.014+0000 7f8d7dc58640 1 librados: starting objecter 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.014+0000 7f8d7dc58640 1 Processor -- start 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.015+0000 7f8d7dc58640 1 -- start start 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.015+0000 7f8d7dc58640 1 librados: setting wanted keys 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.015+0000 7f8d7dc58640 1 librados: calling monclient init 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.015+0000 7f8d7dc58640 10 monclient: init 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.015+0000 7f8d7dc58640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 10 monclient(hunting): picked mon.noname-b con 0x7f8d70030660 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 10 monclient(hunting): picked mon.noname-c con 0x7f8d700389e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 10 monclient(hunting): picked mon.noname-a con 0x7f8d7002d150 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7002d670 con 0x7f8d7002d150 2024-03-29T11:37:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7002f550 con 0x7f8d70030660 2024-03-29T11:37:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700c7a70 con 0x7f8d700389e0 2024-03-29T11:37:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.016+0000 7f8d7dc58640 10 monclient(hunting): authenticate will time out at 1007.280754s 2024-03-29T11:37:31.021 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.021+0000 7f8d7d457640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f8d7002d150 0x7f8d701680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:55475/0 (socket says 172.21.15.33:55475) 2024-03-29T11:37:31.021 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.021+0000 7f8d7d457640 1 -- 172.21.15.33:0/3650820637 learned_addr learned my addr 172.21.15.33:0/3650820637 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:31.023 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.022+0000 7f8d757fa640 10 client.?.objecter ms_handle_connect 0x7f8d7002d150 2024-03-29T11:37:31.023 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.022+0000 7f8d757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:31.024 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 10 client.?.objecter ms_handle_connect 0x7f8d700389e0 2024-03-29T11:37:31.024 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:31.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838200850 0 0) 0x7f8d700c7a70 con 0x7f8d700389e0 2024-03-29T11:37:31.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:37:31.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.024+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c004eb0 con 0x7f8d700389e0 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3869101414 0 0) 0x7f8d5c004eb0 con 0x7f8d700389e0 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d5c005300 con 0x7f8d700389e0 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085094250 0 0) 0x7f8d7002d670 con 0x7f8d7002d150 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c005760 con 0x7f8d7002d150 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8d64001790 con 0x7f8d700389e0 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): dump: 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:31.027 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.025+0000 7f8d757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:31.028 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.026+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3354714029 0 0) 0x7f8d5c005760 con 0x7f8d7002d150 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d7002d670 con 0x7f8d7002d150 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2185689475 0 0) 0x7f8d5c005300 con 0x7f8d700389e0 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 >> v1:172.21.15.153:6789/0 conn(0x7f8d70030660 legacy=0x7f8d7002ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 >> v1:172.21.15.33:6789/0 conn(0x7f8d7002d150 legacy=0x7f8d701680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 1 monclient: found mon.c 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700c0aa0 con 0x7f8d700389e0 2024-03-29T11:37:31.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 10 monclient: _renew_subs 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d700c82c0 con 0x7f8d700389e0 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 10 monclient: _renew_subs 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d701601a0 con 0x7f8d700389e0 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d7dc58640 1 librados: init done 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d64001d60 con 0x7f8d700389e0 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.027+0000 7f8d76ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:31.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.028+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8d64001790 con 0x7f8d700389e0 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.028+0000 7f8d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.028+0000 7f8d757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.028+0000 7f8d757fa640 10 monclient: dump: 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:31.033 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:31.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f8d64019520 con 0x7f8d700389e0 2024-03-29T11:37:31.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d7dc58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:31.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d7dc58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:31.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d7dc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:31.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58005080 con 0x7f8d700389e0 2024-03-29T11:37:31.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f8d64053290 con 0x7f8d700389e0 2024-03-29T11:37:31.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d757fa640 10 client.4352.objecter ms_dispatch 0x7f8d78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:31.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d757fa640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:31.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d757fa640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:31.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.033+0000 7f8d757fa640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:37:31.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.036+0000 7f8d7cc56640 10 monclient: get_auth_request con 0x7f8d5c06e250 auth_method 0 2024-03-29T11:37:31.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.039+0000 7f8d757fa640 10 client.4352.objecter ms_handle_connect 0x7f8d5c06e250 2024-03-29T11:37:31.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.041+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d64052960 con 0x7f8d700389e0 2024-03-29T11:37:31.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.042+0000 7f8d757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:31.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.042+0000 7f8d757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:31.391 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.390+0000 7f8d7dc58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:31.391 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.390+0000 7f8d7dc58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:31.391 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.390+0000 7f8d7dc58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:31.391 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.390+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d58005330 con 0x7f8d700389e0 2024-03-29T11:37:31.391 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.391+0000 7f8d757fa640 1 -- 172.21.15.33:0/3650820637 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f8d64052960 con 0x7f8d700389e0 2024-03-29T11:37:31.391 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.391+0000 7f8d757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:31.392 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.391+0000 7f8d757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:31.393 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:31.393 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 10 librados: watch_flush enter 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 10 librados: watch_flush exit 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 >> v1:172.21.15.153:6816/2737028074 conn(0x7f8d5c06e250 legacy=0x7f8d5c070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 10 monclient: shutdown 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.394+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 >> v1:172.21.15.33:6790/0 conn(0x7f8d700389e0 legacy=0x7f8d700c0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:31.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.395+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 shutdown_connections 2024-03-29T11:37:31.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.395+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 >> 172.21.15.33:0/3650820637 conn(0x7f8d70005c00 msgr2=0x7f8d70034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:31.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.396+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 shutdown_connections 2024-03-29T11:37:31.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.396+0000 7f8d7dc58640 1 -- 172.21.15.33:0/3650820637 wait complete. 2024-03-29T11:37:31.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:31.396+0000 7f8d7dc58640 1 librados: shutdown 2024-03-29T11:37:31.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:38.221 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:38.370 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.370+0000 7f65915a8640 10 monclient: get_monmap_and_config 2024-03-29T11:37:38.370 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.370+0000 7f65915a8640 10 monclient: build_initial_monmap 2024-03-29T11:37:38.370 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.370+0000 7f65915a8640 10 monclient: monmap: 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:38.371323+0000 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:38.371323+0000 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:38.371 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.371+0000 7f65915a8640 1 Processor -- start 2024-03-29T11:37:38.372 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.371+0000 7f65915a8640 1 -- start start 2024-03-29T11:37:38.372 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.371+0000 7f65915a8640 10 monclient: init 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 10 monclient(hunting): picked mon.noname-b con 0x7f658c0a9750 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 10 monclient(hunting): picked mon.noname-c con 0x7f658c149890 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 10 monclient(hunting): picked mon.noname-a con 0x7f658c0afa30 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c0b00c0 con 0x7f658c0afa30 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c1464a0 con 0x7f658c149890 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c0ab280 con 0x7f658c0a9750 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:38.375 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.372+0000 7f65915a8640 10 monclient(hunting): authenticate will time out at 1014.636389s 2024-03-29T11:37:38.376 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.377+0000 7f658a7fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f658c149890 0x7f658c149da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:41691/0 (socket says 172.21.15.33:41691) 2024-03-29T11:37:38.376 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.377+0000 7f658a7fc640 1 -- 172.21.15.33:0/3502662088 learned_addr learned my addr 172.21.15.33:0/3502662088 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:38.378 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.378+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622902434 0 0) 0x7f658c1464a0 con 0x7f658c149890 2024-03-29T11:37:38.378 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.379+0000 7f6589ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.378 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.379+0000 7f6589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.378 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.379+0000 7f6589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.378 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.379+0000 7f6589ffb640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:37:38.378 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.379+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6574003520 con 0x7f658c149890 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346758987 0 0) 0x7f658c0b00c0 con 0x7f658c0afa30 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c1464a0 con 0x7f658c0afa30 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3017639529 0 0) 0x7f6574003520 con 0x7f658c149890 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 >> v1:172.21.15.33:6790/0 conn(0x7f658c0a9750 legacy=0x7f658c1469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 >> v1:172.21.15.33:6789/0 conn(0x7f658c0afa30 legacy=0x7f658c151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 monclient: found mon.noname-c 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c1470b0 con 0x7f658c149890 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f6584001990 con 0x7f658c149890 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: dump: 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.383 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f65915a8640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f65915a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6584001bb0 con 0x7f658c149890 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 1 -- 172.21.15.33:0/3502662088 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f6584001f40 con 0x7f658c149890 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.382+0000 7f6589ffb640 10 monclient: dump: 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:38.385 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:38.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.383+0000 7f65915a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:38.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.383+0000 7f65915a8640 4 set_mon_vals no callback set 2024-03-29T11:37:38.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.384+0000 7f65915a8640 10 monclient: shutdown 2024-03-29T11:37:38.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.384+0000 7f65915a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:38.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.384+0000 7f65915a8640 1 -- 172.21.15.33:0/3502662088 >> v1:172.21.15.153:6789/0 conn(0x7f658c149890 legacy=0x7f658c149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.388+0000 7f65915a8640 1 -- 172.21.15.33:0/3502662088 shutdown_connections 2024-03-29T11:37:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.388+0000 7f65915a8640 1 -- 172.21.15.33:0/3502662088 >> 172.21.15.33:0/3502662088 conn(0x7f658c0b3a00 msgr2=0x7f658c0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:38.389 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.390+0000 7f65915a8640 1 -- 172.21.15.33:0/3502662088 shutdown_connections 2024-03-29T11:37:38.392 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.391+0000 7f65915a8640 1 -- 172.21.15.33:0/3502662088 wait complete. 2024-03-29T11:37:38.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.392+0000 7f65915a8640 10 monclient: build_initial_monmap 2024-03-29T11:37:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.392+0000 7f65915a8640 10 monclient: monmap: 2024-03-29T11:37:38.399 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.393+0000 7f65915a8640 1 librados: starting msgr at 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.393+0000 7f65915a8640 1 librados: starting objecter 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.393+0000 7f65915a8640 1 Processor -- start 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.393+0000 7f65915a8640 1 -- start start 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.393+0000 7f65915a8640 1 librados: setting wanted keys 2024-03-29T11:37:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.393+0000 7f65915a8640 1 librados: calling monclient init 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.393+0000 7f65915a8640 10 monclient: init 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 10 monclient(hunting): picked mon.noname-a con 0x7f658c0a9750 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 10 monclient(hunting): picked mon.noname-c con 0x7f658c0afa30 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 10 monclient(hunting): picked mon.noname-b con 0x7f658c149890 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c1dec60 con 0x7f658c0a9750 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c137f80 con 0x7f658c149890 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c1de8f0 con 0x7f658c0afa30 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.394+0000 7f65915a8640 10 monclient(hunting): authenticate will time out at 1014.658367s 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.398+0000 7f658a7fc640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f658c0afa30 0x7f658c1df160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:59263/0 (socket says 172.21.15.33:59263) 2024-03-29T11:37:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.398+0000 7f658a7fc640 1 -- 172.21.15.33:0/594320096 learned_addr learned my addr 172.21.15.33:0/594320096 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:38.406 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 client.?.objecter ms_handle_connect 0x7f658c0afa30 2024-03-29T11:37:38.406 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:38.406 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 client.?.objecter ms_handle_connect 0x7f658c0a9750 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993748683 0 0) 0x7f658c1de8f0 con 0x7f658c0afa30 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578004eb0 con 0x7f658c0afa30 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 client.?.objecter ms_handle_connect 0x7f658c149890 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806134699 0 0) 0x7f658c1dec60 con 0x7f658c0a9750 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c1de8f0 con 0x7f658c0a9750 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502514122 0 0) 0x7f658c137f80 con 0x7f658c149890 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.401+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c1dec60 con 0x7f658c149890 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2593754534 0 0) 0x7f658c1de8f0 con 0x7f658c0a9750 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f658c137f80 con 0x7f658c0a9750 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f657c0015b0 con 0x7f658c0a9750 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 10 monclient(hunting): dump: 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:38.408 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1574930634 0 0) 0x7f658c137f80 con 0x7f658c0a9750 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 >> v1:172.21.15.33:6790/0 conn(0x7f658c0afa30 legacy=0x7f658c1df160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 >> v1:172.21.15.153:6789/0 conn(0x7f658c149890 legacy=0x7f658c1e5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 monclient: found mon.a 2024-03-29T11:37:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.402+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c1448c0 con 0x7f658c0a9750 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65737fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 10 monclient: _renew_subs 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c144ab0 con 0x7f658c0a9750 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 10 monclient: _renew_subs 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c1445e0 con 0x7f658c0a9750 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.403+0000 7f65915a8640 1 librados: init done 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.404+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f657c001a80 con 0x7f658c0a9750 2024-03-29T11:37:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.404+0000 7f65737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.404+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f657c003840 con 0x7f658c0a9750 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.404+0000 7f65737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.404+0000 7f65737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.404+0000 7f65737fe640 10 monclient: dump: 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:38.411 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.404+0000 7f65897fa640 4 set_mon_vals no callback set 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.406+0000 7f65915a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.406+0000 7f65915a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.406+0000 7f65915a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.406+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c005080 con 0x7f658c0a9750 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.406+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f657c019810 con 0x7f658c0a9750 2024-03-29T11:37:38.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.407+0000 7f658a7fc640 10 monclient: get_auth_request con 0x7f657806de30 auth_method 0 2024-03-29T11:37:38.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.407+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f657c083870 con 0x7f658c0a9750 2024-03-29T11:37:38.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.407+0000 7f65737fe640 10 client.4338.objecter ms_dispatch 0x7f658c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:38.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.407+0000 7f65737fe640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:38.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.407+0000 7f65737fe640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:38.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.408+0000 7f65737fe640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-29T11:37:38.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.408+0000 7f65737fe640 10 client.4338.objecter ms_handle_connect 0x7f657806de30 2024-03-29T11:37:38.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.417+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f657c04f150 con 0x7f658c0a9750 2024-03-29T11:37:38.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.417+0000 7f65737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:38.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.417+0000 7f65737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:38.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.736+0000 7f65915a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:38.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.736+0000 7f65915a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:38.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.736+0000 7f65915a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:37:38.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.736+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f656c005330 con 0x7f658c0a9750 2024-03-29T11:37:38.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.737+0000 7f65737fe640 1 -- 172.21.15.33:0/594320096 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f657c051080 con 0x7f658c0a9750 2024-03-29T11:37:38.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.737+0000 7f65737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:38.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.737+0000 7f65737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:38.737 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:38.737 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:38.740 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.740+0000 7f65915a8640 10 librados: watch_flush enter 2024-03-29T11:37:38.740 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 10 librados: watch_flush exit 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 10 client.4338.objecter successfully canceled tick 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 >> v1:172.21.15.153:6816/2737028074 conn(0x7f657806de30 legacy=0x7f65780702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 10 monclient: shutdown 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 >> v1:172.21.15.33:6789/0 conn(0x7f658c0a9750 legacy=0x7f658c144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 shutdown_connections 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 >> 172.21.15.33:0/594320096 conn(0x7f658c0b3a00 msgr2=0x7f658c0ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 shutdown_connections 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 1 -- 172.21.15.33:0/594320096 wait complete. 2024-03-29T11:37:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:38.741+0000 7f65915a8640 1 librados: shutdown 2024-03-29T11:37:38.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:45.567 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.715+0000 7fd91ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:37:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.715+0000 7fd91ffff640 10 monclient: build_initial_monmap 2024-03-29T11:37:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.715+0000 7fd91ffff640 10 monclient: monmap: 2024-03-29T11:37:45.715 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:45.715 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:45.715 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:45.716009+0000 2024-03-29T11:37:45.715 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:45.716009+0000 2024-03-29T11:37:45.716 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:45.716 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:45.716 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:45.716 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:45.716 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:45.716 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.716+0000 7fd91ffff640 1 Processor -- start 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.716+0000 7fd91ffff640 1 -- start start 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.716+0000 7fd91ffff640 10 monclient: init 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fd9200b2800 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fd9200a9c00 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fd9200b38c0 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd920141eb0 con 0x7fd9200b38c0 2024-03-29T11:37:45.719 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200b22d0 con 0x7fd9200b2800 2024-03-29T11:37:45.719 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd920142d30 con 0x7fd9200a9c00 2024-03-29T11:37:45.719 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:45.719 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.717+0000 7fd91ffff640 10 monclient(hunting): authenticate will time out at 1021.981520s 2024-03-29T11:37:45.721 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.721+0000 7fd917fff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fd9200a9c00 0x7fd92013bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:45863/0 (socket says 172.21.15.33:45863) 2024-03-29T11:37:45.721 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.721+0000 7fd917fff640 1 -- 172.21.15.33:0/3435906775 learned_addr learned my addr 172.21.15.33:0/3435906775 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.722+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000139096 0 0) 0x7fd920142d30 con 0x7fd9200a9c00 2024-03-29T11:37:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.722+0000 7fd91e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.722+0000 7fd91e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.722+0000 7fd91e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.722+0000 7fd91e7fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:37:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.722+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c003520 con 0x7fd9200a9c00 2024-03-29T11:37:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813193679 0 0) 0x7fd90c003520 con 0x7fd9200a9c00 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 >> v1:172.21.15.153:6789/0 conn(0x7fd9200b2800 legacy=0x7fd920142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 >> v1:172.21.15.33:6789/0 conn(0x7fd9200b38c0 legacy=0x7fd92014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200aaf90 con 0x7fd9200a9c00 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd910001690 con 0x7fd9200a9c00 2024-03-29T11:37:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:45.729 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: dump: 2024-03-29T11:37:45.729 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:45.729 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:45.729 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.730 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.730 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.730 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:45.730 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:45.730 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:45.730 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:45.730 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:45.731 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91ffff640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:37:45.731 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:45.731 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9100018b0 con 0x7fd9200a9c00 2024-03-29T11:37:45.731 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:45.731 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 1 -- 172.21.15.33:0/3435906775 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd910001c40 con 0x7fd9200a9c00 2024-03-29T11:37:45.731 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.731 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.723+0000 7fd91e7fc640 10 monclient: dump: 2024-03-29T11:37:45.732 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:45.732 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:45.732 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.732 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.732 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.732 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:45.733 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:45.733 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:45.733 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:45.733 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.727+0000 7fd91ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.727+0000 7fd91ffff640 4 set_mon_vals no callback set 2024-03-29T11:37:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.728+0000 7fd91ffff640 10 monclient: shutdown 2024-03-29T11:37:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.728+0000 7fd91ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.728+0000 7fd91ffff640 1 -- 172.21.15.33:0/3435906775 >> v1:172.21.15.33:6790/0 conn(0x7fd9200a9c00 legacy=0x7fd92013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.728+0000 7fd91ffff640 1 -- 172.21.15.33:0/3435906775 shutdown_connections 2024-03-29T11:37:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.728+0000 7fd91ffff640 1 -- 172.21.15.33:0/3435906775 >> 172.21.15.33:0/3435906775 conn(0x7fd920066a00 msgr2=0x7fd920066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.730+0000 7fd91ffff640 1 -- 172.21.15.33:0/3435906775 shutdown_connections 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.731+0000 7fd91ffff640 1 -- 172.21.15.33:0/3435906775 wait complete. 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 10 monclient: build_initial_monmap 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 10 monclient: monmap: 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:45.739 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:45.740 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:45.740 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:45.740 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:45.740 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:45.740 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:45.740 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:45.740 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 1 librados: starting msgr at 2024-03-29T11:37:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 1 librados: starting objecter 2024-03-29T11:37:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 1 Processor -- start 2024-03-29T11:37:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 1 -- start start 2024-03-29T11:37:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 1 librados: setting wanted keys 2024-03-29T11:37:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 1 librados: calling monclient init 2024-03-29T11:37:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.732+0000 7fd91ffff640 10 monclient: init 2024-03-29T11:37:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fd9200a9c00 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fd9200b2800 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fd9200b38c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92006fb40 con 0x7fd9200b2800 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92006fdb0 con 0x7fd9200a9c00 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd920070090 con 0x7fd9200b38c0 2024-03-29T11:37:45.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:45.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.733+0000 7fd91ffff640 10 monclient(hunting): authenticate will time out at 1021.997252s 2024-03-29T11:37:45.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.739+0000 7fd91effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd9200a9c00 0x7fd9200704b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:39463/0 (socket says 172.21.15.33:39463) 2024-03-29T11:37:45.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.739+0000 7fd91effd640 1 -- 172.21.15.33:0/3053491335 learned_addr learned my addr 172.21.15.33:0/3053491335 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:45.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 client.?.objecter ms_handle_connect 0x7fd9200a9c00 2024-03-29T11:37:45.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:45.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 client.?.objecter ms_handle_connect 0x7fd9200b2800 2024-03-29T11:37:45.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:45.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226216159 0 0) 0x7fd92006fb40 con 0x7fd9200b2800 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.741+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd900004eb0 con 0x7fd9200b2800 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.742+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1849549887 0 0) 0x7fd900004eb0 con 0x7fd9200b2800 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.742+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd92006fb40 con 0x7fd9200b2800 2024-03-29T11:37:45.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.742+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd910001590 con 0x7fd9200b2800 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.742+0000 7fd9177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.742+0000 7fd9177fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.742+0000 7fd9177fe640 10 monclient(hunting): dump: 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.753 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.742+0000 7fd9177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.743+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294701983 0 0) 0x7fd92006fdb0 con 0x7fd9200a9c00 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.743+0000 7fd9177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.743+0000 7fd9177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.754 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.743+0000 7fd9177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.743+0000 7fd9177fe640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:37:45.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.744+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd900004eb0 con 0x7fd9200a9c00 2024-03-29T11:37:45.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.744+0000 7fd9177fe640 10 client.?.objecter ms_handle_connect 0x7fd9200b38c0 2024-03-29T11:37:45.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.744+0000 7fd9177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:45.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.744+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555564875 0 0) 0x7fd920070090 con 0x7fd9200b38c0 2024-03-29T11:37:45.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.744+0000 7fd9177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.744+0000 7fd9177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.744+0000 7fd9177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.745+0000 7fd9177fe640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.745+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92006fdb0 con 0x7fd9200b38c0 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.745+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3069479160 0 0) 0x7fd900004eb0 con 0x7fd9200a9c00 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.745+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd920070090 con 0x7fd9200a9c00 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.745+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd9080017b0 con 0x7fd9200a9c00 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.745+0000 7fd9177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.756 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 10 monclient(hunting): dump: 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:45.757 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:45.758 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:45.758 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:45.758 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:45.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 522185372 0 0) 0x7fd92006fdb0 con 0x7fd9200b38c0 2024-03-29T11:37:45.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd900004eb0 con 0x7fd9200b38c0 2024-03-29T11:37:45.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 684230447 0 0) 0x7fd920070090 con 0x7fd9200a9c00 2024-03-29T11:37:45.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 >> v1:172.21.15.33:6790/0 conn(0x7fd9200b38c0 legacy=0x7fd9201e63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 >> v1:172.21.15.33:6789/0 conn(0x7fd9200b2800 legacy=0x7fd92013da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 1 monclient: found mon.b 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92013f3b0 con 0x7fd9200a9c00 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:45.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:45.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.746+0000 7fd9177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:45.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.747+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd908001cc0 con 0x7fd9200a9c00 2024-03-29T11:37:45.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.747+0000 7fd9177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:45.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.747+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd908004aa0 con 0x7fd9200a9c00 2024-03-29T11:37:45.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.747+0000 7fd9177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.747+0000 7fd9177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:45.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.747+0000 7fd9177fe640 10 monclient: dump: 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:45.761 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 10 monclient: _renew_subs 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92013f600 con 0x7fd9200a9c00 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:45.762 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 10 monclient: _renew_subs 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92006e2e0 con 0x7fd9200a9c00 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.748+0000 7fd91ffff640 1 librados: init done 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.749+0000 7fd91dffb640 4 set_mon_vals no callback set 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.750+0000 7fd91ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.750+0000 7fd91ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.750+0000 7fd91ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:45.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.750+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd904005080 con 0x7fd9200a9c00 2024-03-29T11:37:45.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.751+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fd90801aa70 con 0x7fd9200a9c00 2024-03-29T11:37:45.765 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.752+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fd908084890 con 0x7fd9200a9c00 2024-03-29T11:37:45.766 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.752+0000 7fd9177fe640 10 client.4339.objecter ms_dispatch 0x7fd920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:45.766 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.752+0000 7fd917fff640 10 monclient: get_auth_request con 0x7fd90006e050 auth_method 0 2024-03-29T11:37:45.766 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.752+0000 7fd9177fe640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:45.766 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.752+0000 7fd9177fe640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:45.766 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.752+0000 7fd9177fe640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:37:45.766 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.753+0000 7fd9177fe640 10 client.4339.objecter ms_handle_connect 0x7fd90006e050 2024-03-29T11:37:45.766 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.761+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9080501d0 con 0x7fd9200a9c00 2024-03-29T11:37:45.767 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.761+0000 7fd9177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:45.767 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:45.761+0000 7fd9177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:46.088 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.088+0000 7fd91ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:46.088 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.088+0000 7fd91ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:46.088 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.088+0000 7fd91ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:37:46.088 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.088+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd904005330 con 0x7fd9200a9c00 2024-03-29T11:37:46.089 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.089+0000 7fd9177fe640 1 -- 172.21.15.33:0/3053491335 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fd908083f40 con 0x7fd9200a9c00 2024-03-29T11:37:46.089 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.089+0000 7fd9177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:46.089 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.089+0000 7fd9177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:46.090 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:46.090 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:46.093 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.093+0000 7fd91ffff640 10 librados: watch_flush enter 2024-03-29T11:37:46.093 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.093+0000 7fd91ffff640 10 librados: watch_flush exit 2024-03-29T11:37:46.093 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.093+0000 7fd91ffff640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:37:46.093 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.093+0000 7fd91ffff640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:37:46.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.096+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 >> v1:172.21.15.153:6816/2737028074 conn(0x7fd90006e050 legacy=0x7fd900070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:46.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.096+0000 7fd91ffff640 10 monclient: shutdown 2024-03-29T11:37:46.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.096+0000 7fd91ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:46.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.096+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 >> v1:172.21.15.153:6789/0 conn(0x7fd9200a9c00 legacy=0x7fd9200704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:46.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.097+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 shutdown_connections 2024-03-29T11:37:46.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.097+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 >> 172.21.15.33:0/3053491335 conn(0x7fd920066a00 msgr2=0x7fd920140470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:46.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.097+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 shutdown_connections 2024-03-29T11:37:46.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.097+0000 7fd91ffff640 1 -- 172.21.15.33:0/3053491335 wait complete. 2024-03-29T11:37:46.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:46.098+0000 7fd91ffff640 1 librados: shutdown 2024-03-29T11:37:46.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:52.923 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:53.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.070+0000 7f086f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:53.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.070+0000 7f086f59e640 10 monclient: build_initial_monmap 2024-03-29T11:37:53.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.070+0000 7f086f59e640 10 monclient: monmap: 2024-03-29T11:37:53.070 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:53.070 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:53.070 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:37:53.071201+0000 2024-03-29T11:37:53.070 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:37:53.071201+0000 2024-03-29T11:37:53.071 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:53.071 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:53.071 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:53.071 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:37:53.071 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:37:53.071 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:53.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.072+0000 7f086f59e640 1 Processor -- start 2024-03-29T11:37:53.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.072+0000 7f086f59e640 1 -- start start 2024-03-29T11:37:53.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.072+0000 7f086f59e640 10 monclient: init 2024-03-29T11:37:53.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:53.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:53.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f08680389e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0868038dc0 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0868030660 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08680c54c0 con 0x7f08680389e0 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0868036430 con 0x7f0868038dc0 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0868039840 con 0x7f0868030660 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086f59e640 10 monclient(hunting): authenticate will time out at 1029.337374s 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086ed9d640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f0868030660 0x7f08680d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:56768/0 (socket says 172.21.15.33:56768) 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.073+0000 7f086ed9d640 1 -- 172.21.15.33:0/499281560 learned_addr learned my addr 172.21.15.33:0/499281560 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036434953 0 0) 0x7f0868039840 con 0x7f0868030660 2024-03-29T11:37:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0858003520 con 0x7f0868030660 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711707488 0 0) 0x7f0858003520 con 0x7f0868030660 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.076+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 >> v1:172.21.15.153:6789/0 conn(0x7f0868038dc0 legacy=0x7f08680cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.077+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 >> v1:172.21.15.33:6789/0 conn(0x7f08680389e0 legacy=0x7f0868033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.077+0000 7f086d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.077+0000 7f086d59a640 1 monclient: found mon.noname-b 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.077+0000 7f086d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.077+0000 7f086d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.077+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0868035690 con 0x7f0868030660 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086d59a640 10 monclient: _finish_auth 0 2024-03-29T11:37:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0860001690 con 0x7f0868030660 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086f59e640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086d59a640 10 monclient: dump: 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:53.080 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:53.081 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:53.081 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.078+0000 7f086f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0860001920 con 0x7f0868030660 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086d59a640 1 -- 172.21.15.33:0/499281560 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0860001cb0 con 0x7f0868030660 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086d59a640 10 monclient: dump: 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.082 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.080+0000 7f086f59e640 4 set_mon_vals no callback set 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.081+0000 7f086f59e640 10 monclient: shutdown 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.081+0000 7f086f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.081+0000 7f086f59e640 1 -- 172.21.15.33:0/499281560 >> v1:172.21.15.33:6790/0 conn(0x7f0868030660 legacy=0x7f08680d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.081+0000 7f086f59e640 1 -- 172.21.15.33:0/499281560 shutdown_connections 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.081+0000 7f086f59e640 1 -- 172.21.15.33:0/499281560 >> 172.21.15.33:0/499281560 conn(0x7f0868005c00 msgr2=0x7f0868006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.081+0000 7f086f59e640 1 -- 172.21.15.33:0/499281560 shutdown_connections 2024-03-29T11:37:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.081+0000 7f086f59e640 1 -- 172.21.15.33:0/499281560 wait complete. 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.082+0000 7f086f59e640 10 monclient: build_initial_monmap 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.082+0000 7f086f59e640 10 monclient: monmap: 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:37:53.084 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.083+0000 7f086f59e640 1 librados: starting msgr at 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.083+0000 7f086f59e640 1 librados: starting objecter 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.083+0000 7f086f59e640 1 Processor -- start 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.083+0000 7f086f59e640 1 -- start start 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.083+0000 7f086f59e640 1 librados: setting wanted keys 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.083+0000 7f086f59e640 1 librados: calling monclient init 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.083+0000 7f086f59e640 10 monclient: init 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:53.085 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0868030660 addr v1:172.21.15.153:6789/0 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f08680389e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f086802fe80 addr v1:172.21.15.33:6789/0 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086800ae60 con 0x7f086802fe80 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0868009980 con 0x7f0868030660 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086802f350 con 0x7f08680389e0 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:53.086 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.084+0000 7f086f59e640 10 monclient(hunting): authenticate will time out at 1029.348037s 2024-03-29T11:37:53.087 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.088+0000 7f086dd9b640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f08680389e0 0x7f0868164ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:43134/0 (socket says 172.21.15.33:43134) 2024-03-29T11:37:53.088 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.088+0000 7f086dd9b640 1 -- 172.21.15.33:0/3121980792 learned_addr learned my addr 172.21.15.33:0/3121980792 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:37:53.090 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.090+0000 7f085effd640 10 client.?.objecter ms_handle_connect 0x7f08680389e0 2024-03-29T11:37:53.090 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 10 client.?.objecter ms_handle_connect 0x7f0868030660 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757694238 0 0) 0x7f086802f350 con 0x7f08680389e0 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.091+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0850004eb0 con 0x7f08680389e0 2024-03-29T11:37:53.095 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215259551 0 0) 0x7f0868009980 con 0x7f0868030660 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086802f350 con 0x7f0868030660 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 client.?.objecter ms_handle_connect 0x7f086802fe80 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715692825 0 0) 0x7f086800ae60 con 0x7f086802fe80 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.092+0000 7f085effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0868009980 con 0x7f086802fe80 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1012984163 0 0) 0x7f0850004eb0 con 0x7f08680389e0 2024-03-29T11:37:53.096 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f086800ae60 con 0x7f08680389e0 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f08640017d0 con 0x7f08680389e0 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 10 monclient(hunting): dump: 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:53.097 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984254990 0 0) 0x7f0868009980 con 0x7f086802fe80 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.093+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0850004eb0 con 0x7f086802fe80 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3616569001 0 0) 0x7f086800ae60 con 0x7f08680389e0 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 >> v1:172.21.15.153:6789/0 conn(0x7f0868030660 legacy=0x7f0868164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 >> v1:172.21.15.33:6789/0 conn(0x7f086802fe80 legacy=0x7f086816ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 1 monclient: found mon.c 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086802f4e0 con 0x7f08680389e0 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 10 monclient: _finish_auth 0 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f085effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f086f59e640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:37:53.098 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f086f59e640 10 monclient: _renew_subs 2024-03-29T11:37:53.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f086f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:53.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.094+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086802d150 con 0x7f08680389e0 2024-03-29T11:37:53.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.095+0000 7f086f59e640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:53.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.095+0000 7f086f59e640 10 monclient: _renew_subs 2024-03-29T11:37:53.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.095+0000 7f086f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:53.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.095+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f086816d180 con 0x7f08680389e0 2024-03-29T11:37:53.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.095+0000 7f086f59e640 1 librados: init done 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0864001ce0 con 0x7f08680389e0 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f085effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0864004a00 con 0x7f08680389e0 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f085effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f085effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f085effd640 10 monclient: dump: 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.101 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f086cd99640 4 set_mon_vals no callback set 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.096+0000 7f086f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.097+0000 7f086f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.097+0000 7f086f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.097+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0854005080 con 0x7f08680389e0 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.097+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f086401a9d0 con 0x7f08680389e0 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.098+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f0864084980 con 0x7f08680389e0 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.099+0000 7f085effd640 10 client.4367.objecter ms_dispatch 0x7f0870000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:53.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.099+0000 7f086e59c640 10 monclient: get_auth_request con 0x7f085006de30 auth_method 0 2024-03-29T11:37:53.103 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.099+0000 7f085effd640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:53.103 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.099+0000 7f085effd640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:53.103 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.099+0000 7f085effd640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T11:37:53.103 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.099+0000 7f085effd640 10 client.4367.objecter ms_handle_connect 0x7f085006de30 2024-03-29T11:37:53.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.107+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0864050090 con 0x7f08680389e0 2024-03-29T11:37:53.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.107+0000 7f085effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:53.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.107+0000 7f085effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:53.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.449+0000 7f086f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:53.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.449+0000 7f086f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:53.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.449+0000 7f086f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:37:53.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.449+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0854005330 con 0x7f08680389e0 2024-03-29T11:37:53.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.451+0000 7f085effd640 1 -- 172.21.15.33:0/3121980792 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f0864052190 con 0x7f08680389e0 2024-03-29T11:37:53.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.451+0000 7f085effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:53.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.451+0000 7f085effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:53.451 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:37:53.452 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:53.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.456+0000 7f086f59e640 10 librados: watch_flush enter 2024-03-29T11:37:53.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.456+0000 7f086f59e640 10 librados: watch_flush exit 2024-03-29T11:37:53.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.456+0000 7f086f59e640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T11:37:53.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.456+0000 7f086f59e640 10 client.4367.objecter successfully canceled tick 2024-03-29T11:37:53.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.457+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 >> v1:172.21.15.153:6816/2737028074 conn(0x7f085006de30 legacy=0x7f08500702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.457+0000 7f086f59e640 10 monclient: shutdown 2024-03-29T11:37:53.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.457+0000 7f086f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:53.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.457+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 >> v1:172.21.15.33:6790/0 conn(0x7f08680389e0 legacy=0x7f0868164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:53.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.457+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 shutdown_connections 2024-03-29T11:37:53.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.457+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 >> 172.21.15.33:0/3121980792 conn(0x7f0868005c00 msgr2=0x7f0868034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:53.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.458+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 shutdown_connections 2024-03-29T11:37:53.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.458+0000 7f086f59e640 1 -- 172.21.15.33:0/3121980792 wait complete. 2024-03-29T11:37:53.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:37:53.459+0000 7f086f59e640 1 librados: shutdown 2024-03-29T11:37:53.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:00.282 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.430+0000 7fa8eb050640 10 monclient: get_monmap_and_config 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.430+0000 7fa8eb050640 10 monclient: build_initial_monmap 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.430+0000 7fa8eb050640 10 monclient: monmap: 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:00.431192+0000 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:00.431192+0000 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:00.432 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:00.433 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:00.433 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:00.433 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:00.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.434+0000 7fa8eb050640 1 Processor -- start 2024-03-29T11:38:00.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.434+0000 7fa8eb050640 1 -- start start 2024-03-29T11:38:00.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.434+0000 7fa8eb050640 10 monclient: init 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e40a9930 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e4149dc0 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e414ff40 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e40aae60 con 0x7fa8e414ff40 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e40b4310 con 0x7fa8e40a9930 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e40afdd0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.435+0000 7fa8eb050640 10 monclient(hunting): authenticate will time out at 1036.699035s 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.437+0000 7fa8e3fff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fa8e4149dc0 0x7fa8e414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:47526/0 (socket says 172.21.15.33:47526) 2024-03-29T11:38:00.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.437+0000 7fa8e3fff640 1 -- 172.21.15.33:0/1162109243 learned_addr learned my addr 172.21.15.33:0/1162109243 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:00.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667371511 0 0) 0x7fa8e40afdd0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:38:00.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8cc003520 con 0x7fa8e4149dc0 2024-03-29T11:38:00.440 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330804580 0 0) 0x7fa8e40aae60 con 0x7fa8e414ff40 2024-03-29T11:38:00.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:38:00.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e40afdd0 con 0x7fa8e414ff40 2024-03-29T11:38:00.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3931502741 0 0) 0x7fa8cc003520 con 0x7fa8e4149dc0 2024-03-29T11:38:00.442 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 >> v1:172.21.15.153:6789/0 conn(0x7fa8e40a9930 legacy=0x7fa8e4143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.440+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 >> v1:172.21.15.33:6789/0 conn(0x7fa8e414ff40 legacy=0x7fa8e4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 1 monclient: found mon.noname-b 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e414a950 con 0x7fa8e4149dc0 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa8d8001690 con 0x7fa8e4149dc0 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: dump: 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:00.443 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8d8001870 con 0x7fa8e4149dc0 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 1 -- 172.21.15.33:0/1162109243 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa8d8001c80 con 0x7fa8e4149dc0 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.441+0000 7fa8e37fe640 10 monclient: dump: 2024-03-29T11:38:00.444 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.442+0000 7fa8eb050640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.442+0000 7fa8eb050640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:00.445 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.442+0000 7fa8eb050640 4 set_mon_vals no callback set 2024-03-29T11:38:00.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.443+0000 7fa8eb050640 10 monclient: shutdown 2024-03-29T11:38:00.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.443+0000 7fa8eb050640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:00.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.443+0000 7fa8eb050640 1 -- 172.21.15.33:0/1162109243 >> v1:172.21.15.33:6790/0 conn(0x7fa8e4149dc0 legacy=0x7fa8e414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.443+0000 7fa8eb050640 1 -- 172.21.15.33:0/1162109243 shutdown_connections 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.443+0000 7fa8eb050640 1 -- 172.21.15.33:0/1162109243 >> 172.21.15.33:0/1162109243 conn(0x7fa8e40b35e0 msgr2=0x7fa8e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.443+0000 7fa8eb050640 1 -- 172.21.15.33:0/1162109243 shutdown_connections 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.444+0000 7fa8eb050640 1 -- 172.21.15.33:0/1162109243 wait complete. 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.445+0000 7fa8eb050640 10 monclient: build_initial_monmap 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.445+0000 7fa8eb050640 10 monclient: monmap: 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:00.449 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.445+0000 7fa8eb050640 1 librados: starting msgr at 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.445+0000 7fa8eb050640 1 librados: starting objecter 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.445+0000 7fa8eb050640 1 Processor -- start 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.446+0000 7fa8eb050640 1 -- start start 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.446+0000 7fa8eb050640 1 librados: setting wanted keys 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.446+0000 7fa8eb050640 1 librados: calling monclient init 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.446+0000 7fa8eb050640 10 monclient: init 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e40a9930 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e4149dc0 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e414ff40 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e413f660 con 0x7fa8e40a9930 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e40ac900 con 0x7fa8e414ff40 2024-03-29T11:38:00.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e41df8f0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:00.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8eb050640 10 monclient(hunting): authenticate will time out at 1036.711252s 2024-03-29T11:38:00.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8e3fff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fa8e4149dc0 0x7fa8e4148260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:47932/0 (socket says 172.21.15.33:47932) 2024-03-29T11:38:00.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8e3fff640 1 -- 172.21.15.33:0/1441353705 learned_addr learned my addr 172.21.15.33:0/1441353705 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:00.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8e17fa640 10 client.?.objecter ms_handle_connect 0x7fa8e4149dc0 2024-03-29T11:38:00.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.447+0000 7fa8e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:00.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.448+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167819990 0 0) 0x7fa8e41df8f0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.448+0000 7fa8e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:00.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.448+0000 7fa8e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:00.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.448+0000 7fa8e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:00.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.448+0000 7fa8e17fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.448+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d0004eb0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434673325 0 0) 0x7fa8d0004eb0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8e41df8f0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa8d8001c80 con 0x7fa8e4149dc0 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 10 monclient(hunting): dump: 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:00.455 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2035499073 0 0) 0x7fa8e41df8f0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.449+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 >> v1:172.21.15.153:6789/0 conn(0x7fa8e414ff40 legacy=0x7fa8e4148970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.450+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 >> v1:172.21.15.33:6789/0 conn(0x7fa8e40a9930 legacy=0x7fa8e40ac1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.451+0000 7fa8e17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.451+0000 7fa8e17fa640 1 monclient: found mon.c 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.451+0000 7fa8e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.451+0000 7fa8e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.451+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e41dfa80 con 0x7fa8e4149dc0 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.451+0000 7fa8e17fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8e17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:38:00.456 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 10 monclient: _renew_subs 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 1 -- 172.21.15.33:0/1441353705 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e40ad120 con 0x7fa8e4149dc0 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 10 monclient: _renew_subs 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 1 -- 172.21.15.33:0/1441353705 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e41df020 con 0x7fa8e4149dc0 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.452+0000 7fa8eb050640 1 librados: init done 2024-03-29T11:38:00.457 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8d8004bf0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:00.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa8d8004f00 con 0x7fa8e4149dc0 2024-03-29T11:38:00.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:00.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:00.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8caffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:00.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8e2ffd640 4 set_mon_vals no callback set 2024-03-29T11:38:00.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.455+0000 7fa8e17fa640 10 monclient: dump: 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.456+0000 7fa8caffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.456+0000 7fa8caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.456+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e4075f10 con 0x7fa8e4149dc0 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.457+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fa8d801aed0 con 0x7fa8e4149dc0 2024-03-29T11:38:00.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.458+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fa8d8084d60 con 0x7fa8e4149dc0 2024-03-29T11:38:00.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.458+0000 7fa8e17fa640 10 client.4373.objecter ms_dispatch 0x7fa8e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:00.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.458+0000 7fa8e17fa640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:00.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.458+0000 7fa8e17fa640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:00.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.458+0000 7fa8e17fa640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:38:00.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.459+0000 7fa8e8dec640 10 monclient: get_auth_request con 0x7fa8d006dd10 auth_method 0 2024-03-29T11:38:00.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.460+0000 7fa8e17fa640 10 client.4373.objecter ms_handle_connect 0x7fa8d006dd10 2024-03-29T11:38:00.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.466+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8d8050640 con 0x7fa8e4149dc0 2024-03-29T11:38:00.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.466+0000 7fa8e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:00.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.466+0000 7fa8e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:00.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.792+0000 7fa8caffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:00.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.792+0000 7fa8caffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:00.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.792+0000 7fa8caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:00.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.792+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8e40b1b40 con 0x7fa8e4149dc0 2024-03-29T11:38:00.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.793+0000 7fa8e17fa640 1 -- 172.21.15.33:0/1441353705 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fa8d8052570 con 0x7fa8e4149dc0 2024-03-29T11:38:00.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.793+0000 7fa8e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:00.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.793+0000 7fa8e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:00.793 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:00.793 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:00.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.796+0000 7fa8caffd640 10 librados: watch_flush enter 2024-03-29T11:38:00.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.797+0000 7fa8caffd640 10 librados: watch_flush exit 2024-03-29T11:38:00.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.797+0000 7fa8caffd640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:38:00.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.797+0000 7fa8caffd640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:38:00.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.797+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 >> v1:172.21.15.153:6816/2737028074 conn(0x7fa8d006dd10 legacy=0x7fa8d00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.797+0000 7fa8caffd640 10 monclient: shutdown 2024-03-29T11:38:00.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.797+0000 7fa8caffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:00.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.797+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 >> v1:172.21.15.33:6790/0 conn(0x7fa8e4149dc0 legacy=0x7fa8e4148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.798+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 shutdown_connections 2024-03-29T11:38:00.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.798+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 >> 172.21.15.33:0/1441353705 conn(0x7fa8e40b35e0 msgr2=0x7fa8e40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:00.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.798+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 shutdown_connections 2024-03-29T11:38:00.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.798+0000 7fa8caffd640 1 -- 172.21.15.33:0/1441353705 wait complete. 2024-03-29T11:38:00.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:00.798+0000 7fa8caffd640 1 librados: shutdown 2024-03-29T11:38:00.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:07.628 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:07.769 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.768+0000 7f263bb0e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.768+0000 7f263bb0e640 10 monclient: build_initial_monmap 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.768+0000 7f263bb0e640 10 monclient: monmap: 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:07.770362+0000 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:07.770362+0000 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:07.770 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.769+0000 7f263bb0e640 1 Processor -- start 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.769+0000 7f263bb0e640 1 -- start start 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.769+0000 7f263bb0e640 10 monclient: init 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f26341457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f263413f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f26340a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:07.776 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26340aaef0 con 0x7f26341457f0 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2634145bd0 con 0x7f263413f650 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2634145eb0 con 0x7f26340a9d70 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f26398aa640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f26341457f0 0x7f263413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:47950/0 (socket says 172.21.15.33:47950) 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f26398aa640 1 -- 172.21.15.33:0/4182207767 learned_addr learned my addr 172.21.15.33:0/4182207767 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.770+0000 7f263bb0e640 10 monclient(hunting): authenticate will time out at 1044.035396s 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302167849 0 0) 0x7f2634145eb0 con 0x7f26340a9d70 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2620003520 con 0x7f26340a9d70 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531609215 0 0) 0x7f2634145bd0 con 0x7f263413f650 2024-03-29T11:38:07.777 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.771+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2634145eb0 con 0x7f263413f650 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515317679 0 0) 0x7f26340aaef0 con 0x7f26341457f0 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2634145bd0 con 0x7f26341457f0 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1210684829 0 0) 0x7f2634145bd0 con 0x7f26341457f0 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 >> v1:172.21.15.33:6790/0 conn(0x7f26340a9d70 legacy=0x7f263414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 >> v1:172.21.15.153:6789/0 conn(0x7f263413f650 legacy=0x7f26340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient: _finish_hunting 0 2024-03-29T11:38:07.778 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 monclient: found mon.noname-a 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634146340 con 0x7f26341457f0 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient: _finish_auth 0 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient: _check_auth_tickets 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2630001630 con 0x7f26341457f0 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f26388a8640 10 monclient: dump: 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.779 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f263bb0e640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.772+0000 7f263bb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2630001850 con 0x7f26341457f0 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f26388a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f26388a8640 1 -- 172.21.15.33:0/4182207767 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2630001b60 con 0x7f26341457f0 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f26388a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f26388a8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f26388a8640 10 monclient: dump: 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:07.780 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f263bb0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.773+0000 7f263bb0e640 4 set_mon_vals no callback set 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.774+0000 7f263bb0e640 10 monclient: shutdown 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.774+0000 7f263bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.774+0000 7f263bb0e640 1 -- 172.21.15.33:0/4182207767 >> v1:172.21.15.33:6789/0 conn(0x7f26341457f0 legacy=0x7f263413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.774+0000 7f263bb0e640 1 -- 172.21.15.33:0/4182207767 shutdown_connections 2024-03-29T11:38:07.781 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.774+0000 7f263bb0e640 1 -- 172.21.15.33:0/4182207767 >> 172.21.15.33:0/4182207767 conn(0x7f2634066a00 msgr2=0x7f2634066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:07.782 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.774+0000 7f263bb0e640 1 -- 172.21.15.33:0/4182207767 shutdown_connections 2024-03-29T11:38:07.782 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.774+0000 7f263bb0e640 1 -- 172.21.15.33:0/4182207767 wait complete. 2024-03-29T11:38:07.789 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.775+0000 7f263bb0e640 10 monclient: build_initial_monmap 2024-03-29T11:38:07.789 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.775+0000 7f263bb0e640 10 monclient: monmap: 2024-03-29T11:38:07.789 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:07.789 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.776+0000 7f263bb0e640 1 librados: starting msgr at 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.776+0000 7f263bb0e640 1 librados: starting objecter 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.776+0000 7f263bb0e640 1 Processor -- start 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.776+0000 7f263bb0e640 1 -- start start 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.776+0000 7f263bb0e640 1 librados: setting wanted keys 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.776+0000 7f263bb0e640 1 librados: calling monclient init 2024-03-29T11:38:07.790 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.776+0000 7f263bb0e640 10 monclient: init 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.777+0000 7f263bb0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.777+0000 7f263bb0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.777+0000 7f263bb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f26340a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.777+0000 7f263bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f263413f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f26390a9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f263413f650 0x7f2634139570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:54992/0 (socket says 172.21.15.33:54992) 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f26390a9640 1 -- 172.21.15.33:0/1524724963 learned_addr learned my addr 172.21.15.33:0/1524724963 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f263bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f26341457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2634142170 con 0x7f26341457f0 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2634143fc0 con 0x7f263413f650 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2634142c90 con 0x7f26340a9d70 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f263bb0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f263bb0e640 10 monclient(hunting): authenticate will time out at 1044.043048s 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f262a7fc640 10 client.?.objecter ms_handle_connect 0x7f26341457f0 2024-03-29T11:38:07.791 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.778+0000 7f262a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 client.?.objecter ms_handle_connect 0x7f26340a9d70 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 client.?.objecter ms_handle_connect 0x7f263413f650 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479519580 0 0) 0x7f2634143fc0 con 0x7f263413f650 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2618004eb0 con 0x7f263413f650 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129343579 0 0) 0x7f2634142170 con 0x7f26341457f0 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.779+0000 7f262a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.792 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2634143fc0 con 0x7f26341457f0 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560001828 0 0) 0x7f2634142c90 con 0x7f26340a9d70 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2634142170 con 0x7f26340a9d70 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1268361968 0 0) 0x7f2618004eb0 con 0x7f263413f650 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2634142c90 con 0x7f263413f650 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2624001910 con 0x7f263413f650 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): dump: 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:07.793 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384134161 0 0) 0x7f2634143fc0 con 0x7f26341457f0 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2618004eb0 con 0x7f26341457f0 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.780+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f262c001b80 con 0x7f26341457f0 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:07.794 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 10 monclient(hunting): dump: 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573503099 0 0) 0x7f2634142170 con 0x7f26340a9d70 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2634143fc0 con 0x7f26340a9d70 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2384618285 0 0) 0x7f2634142c90 con 0x7f263413f650 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 >> v1:172.21.15.33:6790/0 conn(0x7f26340a9d70 legacy=0x7f26341438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 >> v1:172.21.15.33:6789/0 conn(0x7f26341457f0 legacy=0x7f26341e7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.795 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 1 monclient: found mon.b 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.781+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263413a1c0 con 0x7f263413f650 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f262a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f262a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f262a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 10 monclient: _renew_subs 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2634141c00 con 0x7f263413f650 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 10 monclient: _renew_subs 2024-03-29T11:38:07.796 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2634139c80 con 0x7f263413f650 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.782+0000 7f263bb0e640 1 librados: init done 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.783+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2624003610 con 0x7f263413f650 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.783+0000 7f262a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.783+0000 7f262bfff640 4 set_mon_vals no callback set 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.783+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f26240039a0 con 0x7f263413f650 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.783+0000 7f262a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.783+0000 7f262a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.783+0000 7f262a7fc640 10 monclient: dump: 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:07.797 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.785+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f2624003b00 con 0x7f263413f650 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.786+0000 7f26398aa640 10 monclient: get_auth_request con 0x7f261806e380 auth_method 0 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.786+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f2624084980 con 0x7f263413f650 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.786+0000 7f262a7fc640 10 client.4348.objecter ms_dispatch 0x7f2634000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.786+0000 7f262a7fc640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.786+0000 7f262a7fc640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.786+0000 7f262a7fc640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.787+0000 7f262a7fc640 10 client.4348.objecter ms_handle_connect 0x7f261806e380 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.787+0000 7f263bb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.787+0000 7f263bb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:07.798 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.787+0000 7f263bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:07.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.787+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261c005080 con 0x7f263413f650 2024-03-29T11:38:07.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.797+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2624050260 con 0x7f263413f650 2024-03-29T11:38:07.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.797+0000 7f262a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:07.799 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:07.797+0000 7f262a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:08.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.199+0000 7f263bb0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:08.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.199+0000 7f263bb0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:08.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.199+0000 7f263bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:08.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.199+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f261c005330 con 0x7f263413f650 2024-03-29T11:38:08.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.202+0000 7f262a7fc640 1 -- 172.21.15.33:0/1524724963 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f2624052190 con 0x7f263413f650 2024-03-29T11:38:08.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.202+0000 7f262a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:08.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.202+0000 7f262a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:08.204 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:08.204 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:08.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 10 librados: watch_flush enter 2024-03-29T11:38:08.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 10 librados: watch_flush exit 2024-03-29T11:38:08.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:38:08.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:38:08.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 >> v1:172.21.15.153:6816/2737028074 conn(0x7f261806e380 legacy=0x7f2618070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 10 monclient: shutdown 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.206+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 >> v1:172.21.15.153:6789/0 conn(0x7f263413f650 legacy=0x7f2634139570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.207+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 shutdown_connections 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.207+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 >> 172.21.15.33:0/1524724963 conn(0x7f2634066a00 msgr2=0x7f26341d26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.207+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 shutdown_connections 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.207+0000 7f263bb0e640 1 -- 172.21.15.33:0/1524724963 wait complete. 2024-03-29T11:38:08.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:08.207+0000 7f263bb0e640 1 librados: shutdown 2024-03-29T11:38:08.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:15.031 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:15.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.164+0000 7f7823288640 10 monclient: get_monmap_and_config 2024-03-29T11:38:15.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.164+0000 7f7823288640 10 monclient: build_initial_monmap 2024-03-29T11:38:15.166 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.164+0000 7f7823288640 10 monclient: monmap: 2024-03-29T11:38:15.166 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:15.166 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:15.166338+0000 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:15.166338+0000 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:15.167 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.165+0000 7f7823288640 1 Processor -- start 2024-03-29T11:38:15.170 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.165+0000 7f7823288640 1 -- start start 2024-03-29T11:38:15.170 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.165+0000 7f7823288640 10 monclient: init 2024-03-29T11:38:15.170 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.166+0000 7f7823288640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.166+0000 7f7823288640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.166+0000 7f7823288640 10 monclient(hunting): picked mon.noname-c con 0x7f781c0b0860 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.166+0000 7f7823288640 10 monclient(hunting): picked mon.noname-a con 0x7f781c072450 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7823288640 10 monclient(hunting): picked mon.noname-b con 0x7f781c13ee40 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7821024640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f781c0b0860 0x7f781c0acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:41516/0 (socket says 172.21.15.33:41516) 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7821024640 1 -- 172.21.15.33:0/3158055019 learned_addr learned my addr 172.21.15.33:0/3158055019 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c0ad840 con 0x7f781c072450 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c13c830 con 0x7f781c0b0860 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c13c9c0 con 0x7f781c13ee40 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7823288640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7823288640 10 monclient(hunting): authenticate will time out at 1051.432317s 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52697138 0 0) 0x7f781c13c9c0 con 0x7f781c13ee40 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7813fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7813fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.171 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7813fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7813fff640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.167+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7808003520 con 0x7f781c13ee40 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763152067 0 0) 0x7f781c13c830 con 0x7f781c0b0860 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f781c13c9c0 con 0x7f781c0b0860 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248946436 0 0) 0x7f7808003520 con 0x7f781c13ee40 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 >> v1:172.21.15.153:6789/0 conn(0x7f781c0b0860 legacy=0x7f781c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.172 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.168+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 >> v1:172.21.15.33:6789/0 conn(0x7f781c072450 legacy=0x7f781c13c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 1 monclient: found mon.noname-b 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c13f970 con 0x7f781c13ee40 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7814001a10 con 0x7f781c13ee40 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:15.174 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: dump: 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7823288640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7823288640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7814001b70 con 0x7f781c13ee40 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 1 -- 172.21.15.33:0/3158055019 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7814001f80 con 0x7f781c13ee40 2024-03-29T11:38:15.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7813fff640 10 monclient: dump: 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7823288640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.170+0000 7f7823288640 4 set_mon_vals no callback set 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.171+0000 7f7823288640 10 monclient: shutdown 2024-03-29T11:38:15.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.171+0000 7f7823288640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.171+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 >> v1:172.21.15.33:6790/0 conn(0x7f781c13ee40 legacy=0x7f781c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.171+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 shutdown_connections 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.172+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 >> 172.21.15.33:0/3158055019 conn(0x7f781c06b660 msgr2=0x7f781c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.172+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 shutdown_connections 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.172+0000 7f7823288640 1 -- 172.21.15.33:0/3158055019 wait complete. 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 10 monclient: build_initial_monmap 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 10 monclient: monmap: 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:15.177 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:15.178 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:15.178 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:15.178 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 1 librados: starting msgr at 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 1 librados: starting objecter 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 1 Processor -- start 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 1 -- start start 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 1 librados: setting wanted keys 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 1 librados: calling monclient init 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.173+0000 7f7823288640 10 monclient: init 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.174+0000 7f7823288640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.174+0000 7f7823288640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.174+0000 7f7823288640 10 monclient(hunting): picked mon.noname-b con 0x7f781c072450 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7820823640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f781c0b0860 0x7f781c141c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:40694/0 (socket says 172.21.15.33:40694) 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7821024640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f781c072450 0x7f781c1d6a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:41714/0 (socket says 172.21.15.33:41714) 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7823288640 10 monclient(hunting): picked mon.noname-c con 0x7f781c0b0860 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:15.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7823288640 10 monclient(hunting): picked mon.noname-a con 0x7f781c13ee40 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7823288640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c1431d0 con 0x7f781c13ee40 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7823288640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c1d7110 con 0x7f781c072450 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7823288640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c1d64e0 con 0x7f781c0b0860 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7823288640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7823288640 10 monclient(hunting): authenticate will time out at 1051.440003s 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7820823640 1 -- 172.21.15.33:0/1733141598 learned_addr learned my addr 172.21.15.33:0/1733141598 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7811ffb640 10 client.?.objecter ms_handle_connect 0x7f781c0b0860 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7811ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7811ffb640 10 client.?.objecter ms_handle_connect 0x7f781c072450 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.175+0000 7f7811ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555650496 0 0) 0x7f781c1d7110 con 0x7f781c072450 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:38:15.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7800004eb0 con 0x7f781c072450 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468213066 0 0) 0x7f781c1d64e0 con 0x7f781c0b0860 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78000051d0 con 0x7f781c0b0860 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714831083 0 0) 0x7f78000051d0 con 0x7f781c0b0860 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f781c1d64e0 con 0x7f781c0b0860 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f780c001910 con 0x7f781c0b0860 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:15.185 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.176+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2518310353 0 0) 0x7f781c1d64e0 con 0x7f781c0b0860 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 >> v1:172.21.15.153:6789/0 conn(0x7f781c072450 legacy=0x7f781c1d6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 >> v1:172.21.15.33:6789/0 conn(0x7f781c13ee40 legacy=0x7f781c0b3990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 1 monclient: found mon.c 2024-03-29T11:38:15.186 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c142310 con 0x7f781c0b0860 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.177+0000 7f7811ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 10 monclient: _renew_subs 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c136de0 con 0x7f781c0b0860 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 10 monclient: _renew_subs 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f781c0b40a0 con 0x7f781c0b0860 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.178+0000 7f7823288640 1 librados: init done 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f780c0035f0 con 0x7f781c0b0860 2024-03-29T11:38:15.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f7811ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f780c0039c0 con 0x7f781c0b0860 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f7811ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f7811ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f7811ffb640 10 monclient: dump: 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f78077fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:15.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f78077fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f78077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.179+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc005080 con 0x7f781c0b0860 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.180+0000 7f78137fe640 4 set_mon_vals no callback set 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.181+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f780c003b20 con 0x7f781c0b0860 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.181+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f780c084990 con 0x7f781c0b0860 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.181+0000 7f7811ffb640 10 client.4385.objecter ms_dispatch 0x7f781c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.181+0000 7f7811ffb640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.181+0000 7f7811ffb640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.181+0000 7f7811ffb640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:38:15.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.182+0000 7f7821024640 10 monclient: get_auth_request con 0x7f780006deb0 auth_method 0 2024-03-29T11:38:15.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.182+0000 7f7811ffb640 10 client.4385.objecter ms_handle_connect 0x7f780006deb0 2024-03-29T11:38:15.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.188+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f780c050270 con 0x7f781c0b0860 2024-03-29T11:38:15.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.188+0000 7f7811ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:15.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.188+0000 7f7811ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:15.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.499+0000 7f78077fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:15.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.499+0000 7f78077fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:15.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.499+0000 7f78077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:38:15.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.499+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77fc005330 con 0x7f781c0b0860 2024-03-29T11:38:15.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.500+0000 7f7811ffb640 1 -- 172.21.15.33:0/1733141598 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f780c0521a0 con 0x7f781c0b0860 2024-03-29T11:38:15.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.500+0000 7f7811ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:15.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.500+0000 7f7811ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:15.501 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:15.501 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:15.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 10 librados: watch_flush enter 2024-03-29T11:38:15.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 10 librados: watch_flush exit 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 >> v1:172.21.15.153:6816/2737028074 conn(0x7f780006deb0 legacy=0x7f7800070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 10 monclient: shutdown 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 >> v1:172.21.15.33:6790/0 conn(0x7f781c0b0860 legacy=0x7f781c141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 shutdown_connections 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 >> 172.21.15.33:0/1733141598 conn(0x7f781c06b660 msgr2=0x7f781c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 shutdown_connections 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.503+0000 7f78077fe640 1 -- 172.21.15.33:0/1733141598 wait complete. 2024-03-29T11:38:15.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:15.504+0000 7f78077fe640 1 librados: shutdown 2024-03-29T11:38:15.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:22.324 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:22.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.476+0000 7f37af20f640 10 monclient: get_monmap_and_config 2024-03-29T11:38:22.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.476+0000 7f37af20f640 10 monclient: build_initial_monmap 2024-03-29T11:38:22.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.476+0000 7f37af20f640 10 monclient: monmap: 2024-03-29T11:38:22.477 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:22.477 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:22.477 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:22.477736+0000 2024-03-29T11:38:22.478 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:22.477736+0000 2024-03-29T11:38:22.478 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:22.478 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:22.478 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:22.478 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:22.478 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:22.478 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:22.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.477+0000 7f37af20f640 1 Processor -- start 2024-03-29T11:38:22.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.477+0000 7f37af20f640 1 -- start start 2024-03-29T11:38:22.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.477+0000 7f37af20f640 10 monclient: init 2024-03-29T11:38:22.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:22.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:22.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 10 monclient(hunting): picked mon.noname-b con 0x7f37a806d9d0 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:22.480 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 10 monclient(hunting): picked mon.noname-c con 0x7f37a813e820 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:22.480 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 10 monclient(hunting): picked mon.noname-a con 0x7f37a80af1d0 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:22.480 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a80aed10 con 0x7f37a80af1d0 2024-03-29T11:38:22.480 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a80afd50 con 0x7f37a813e820 2024-03-29T11:38:22.480 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a80ab490 con 0x7f37a806d9d0 2024-03-29T11:38:22.480 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:22.480 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.478+0000 7f37af20f640 10 monclient(hunting): authenticate will time out at 1058.742904s 2024-03-29T11:38:22.481 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.480+0000 7f37a7fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f37a813e820 0x7f37a813ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:49246/0 (socket says 172.21.15.33:49246) 2024-03-29T11:38:22.481 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.480+0000 7f37a7fff640 1 -- 172.21.15.33:0/3617827964 learned_addr learned my addr 172.21.15.33:0/3617827964 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:22.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.482+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300939723 0 0) 0x7f37a80afd50 con 0x7f37a813e820 2024-03-29T11:38:22.483 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.482+0000 7f37a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:22.483 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.482+0000 7f37a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:22.483 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.482+0000 7f37a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:22.483 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.482+0000 7f37a77fe640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:38:22.483 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.482+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3790003520 con 0x7f37a813e820 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667520155 0 0) 0x7f3790003520 con 0x7f37a813e820 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 >> v1:172.21.15.33:6790/0 conn(0x7f37a806d9d0 legacy=0x7f37a80b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 >> v1:172.21.15.33:6789/0 conn(0x7f37a80af1d0 legacy=0x7f37a80af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 1 monclient: found mon.noname-c 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a80ab770 con 0x7f37a813e820 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3798001690 con 0x7f37a813e820 2024-03-29T11:38:22.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37a77fe640 10 monclient: dump: 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37af20f640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:38:22.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.485+0000 7f37af20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:22.490 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.487+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3798001860 con 0x7f37a813e820 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.487+0000 7f37a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.487+0000 7f37a77fe640 1 -- 172.21.15.33:0/3617827964 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3798001cb0 con 0x7f37a813e820 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.487+0000 7f37a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.487+0000 7f37a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.487+0000 7f37a77fe640 10 monclient: dump: 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:22.491 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.488+0000 7f37af20f640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.488+0000 7f37af20f640 4 set_mon_vals no callback set 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.489+0000 7f37af20f640 10 monclient: shutdown 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.489+0000 7f37af20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.489+0000 7f37af20f640 1 -- 172.21.15.33:0/3617827964 >> v1:172.21.15.153:6789/0 conn(0x7f37a813e820 legacy=0x7f37a813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.489+0000 7f37af20f640 1 -- 172.21.15.33:0/3617827964 shutdown_connections 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.489+0000 7f37af20f640 1 -- 172.21.15.33:0/3617827964 >> 172.21.15.33:0/3617827964 conn(0x7f37a806b660 msgr2=0x7f37a806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.490+0000 7f37af20f640 1 -- 172.21.15.33:0/3617827964 shutdown_connections 2024-03-29T11:38:22.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.490+0000 7f37af20f640 1 -- 172.21.15.33:0/3617827964 wait complete. 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 10 monclient: build_initial_monmap 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 10 monclient: monmap: 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:22.494 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 1 librados: starting msgr at 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 1 librados: starting objecter 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 1 Processor -- start 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 1 -- start start 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 1 librados: setting wanted keys 2024-03-29T11:38:22.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 1 librados: calling monclient init 2024-03-29T11:38:22.496 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.493+0000 7f37af20f640 10 monclient: init 2024-03-29T11:38:22.496 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.494+0000 7f37af20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:22.496 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.494+0000 7f37af20f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:22.496 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.494+0000 7f37af20f640 10 monclient(hunting): picked mon.noname-a con 0x7f37a806d9d0 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:22.496 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.494+0000 7f37af20f640 10 monclient(hunting): picked mon.noname-c con 0x7f37a80af1d0 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:22.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.497+0000 7f37af20f640 10 monclient(hunting): picked mon.noname-b con 0x7f37a813e820 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:22.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.497+0000 7f37af20f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a8143050 con 0x7f37a806d9d0 2024-03-29T11:38:22.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.497+0000 7f37af20f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a80b3e30 con 0x7f37a813e820 2024-03-29T11:38:22.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.497+0000 7f37af20f640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a8070a10 con 0x7f37a80af1d0 2024-03-29T11:38:22.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.497+0000 7f37af20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:22.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.497+0000 7f37af20f640 10 monclient(hunting): authenticate will time out at 1058.761770s 2024-03-29T11:38:22.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.498+0000 7f37a7fff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f37a80af1d0 0x7f37a80b27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:60235/0 (socket says 172.21.15.33:60235) 2024-03-29T11:38:22.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.498+0000 7f37a7fff640 1 -- 172.21.15.33:0/3911135178 learned_addr learned my addr 172.21.15.33:0/3911135178 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:22.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.499+0000 7f37a57fa640 10 client.?.objecter ms_handle_connect 0x7f37a80af1d0 2024-03-29T11:38:22.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.499+0000 7f37a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:22.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.499+0000 7f37a57fa640 10 client.?.objecter ms_handle_connect 0x7f37a806d9d0 2024-03-29T11:38:22.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.499+0000 7f37a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:22.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 10 client.?.objecter ms_handle_connect 0x7f37a813e820 2024-03-29T11:38:22.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:22.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010514407 0 0) 0x7f37a8143050 con 0x7f37a806d9d0 2024-03-29T11:38:22.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794004eb0 con 0x7f37a806d9d0 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242639207 0 0) 0x7f37a80b3e30 con 0x7f37a813e820 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.500+0000 7f37a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.501+0000 7f37a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.501+0000 7f37a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.501+0000 7f37a57fa640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:38:22.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.501+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37a8143050 con 0x7f37a813e820 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.501+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2479256572 0 0) 0x7f37a8143050 con 0x7f37a813e820 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.501+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37a80b3e30 con 0x7f37a813e820 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f37a0001a90 con 0x7f37a813e820 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): dump: 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:22.504 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156151275 0 0) 0x7f37a8070a10 con 0x7f37a80af1d0 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37a8143050 con 0x7f37a80af1d0 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2092030109 0 0) 0x7f37a80b3e30 con 0x7f37a813e820 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 >> v1:172.21.15.33:6790/0 conn(0x7f37a80af1d0 legacy=0x7f37a80b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 >> v1:172.21.15.33:6789/0 conn(0x7f37a806d9d0 legacy=0x7f37a806ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 monclient: found mon.b 2024-03-29T11:38:22.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a8070cf0 con 0x7f37a813e820 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 10 monclient: _renew_subs 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a8070fd0 con 0x7f37a813e820 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 10 monclient: _renew_subs 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a8071940 con 0x7f37a813e820 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37af20f640 1 librados: init done 2024-03-29T11:38:22.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a0001bf0 con 0x7f37a813e820 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.502+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f37a0003970 con 0x7f37a813e820 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.503+0000 7f37a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.503+0000 7f37a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.503+0000 7f37a57fa640 10 monclient: dump: 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:22.507 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.505+0000 7f37a6ffd640 4 set_mon_vals no callback set 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.506+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f37a0004bc0 con 0x7f37a813e820 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37af20f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37af20f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37af20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3788005080 con 0x7f37a813e820 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f37a00839f0 con 0x7f37a813e820 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37a57fa640 10 client.4360.objecter ms_dispatch 0x7f37a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37acfab640 10 monclient: get_auth_request con 0x7f379406de60 auth_method 0 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.507+0000 7f37a57fa640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.508+0000 7f37a57fa640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:22.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.508+0000 7f37a57fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:38:22.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.508+0000 7f37a57fa640 10 client.4360.objecter ms_handle_connect 0x7f379406de60 2024-03-29T11:38:22.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.516+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37a004f2d0 con 0x7f37a813e820 2024-03-29T11:38:22.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.516+0000 7f37a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:22.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.516+0000 7f37a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:22.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.856+0000 7f37af20f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:22.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.856+0000 7f37af20f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:22.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.856+0000 7f37af20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:22.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.856+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3788005330 con 0x7f37a813e820 2024-03-29T11:38:22.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.859+0000 7f37a57fa640 1 -- 172.21.15.33:0/3911135178 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f37a0051200 con 0x7f37a813e820 2024-03-29T11:38:22.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.859+0000 7f37a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:22.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.859+0000 7f37a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:22.861 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:22.861 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:22.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.862+0000 7f37af20f640 10 librados: watch_flush enter 2024-03-29T11:38:22.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 10 librados: watch_flush exit 2024-03-29T11:38:22.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:38:22.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:38:22.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 >> v1:172.21.15.153:6816/2737028074 conn(0x7f379406de60 legacy=0x7f3794070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 10 monclient: shutdown 2024-03-29T11:38:22.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:22.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 >> v1:172.21.15.153:6789/0 conn(0x7f37a813e820 legacy=0x7f37a80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 shutdown_connections 2024-03-29T11:38:22.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.863+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 >> 172.21.15.33:0/3911135178 conn(0x7f37a806b660 msgr2=0x7f37a80ac280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:22.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.864+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 shutdown_connections 2024-03-29T11:38:22.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.864+0000 7f37af20f640 1 -- 172.21.15.33:0/3911135178 wait complete. 2024-03-29T11:38:22.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:22.864+0000 7f37af20f640 1 librados: shutdown 2024-03-29T11:38:22.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:29.689 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.845+0000 7f61527e3640 10 monclient: get_monmap_and_config 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.845+0000 7f61527e3640 10 monclient: build_initial_monmap 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.845+0000 7f61527e3640 10 monclient: monmap: 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:29.846660+0000 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:29.846660+0000 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:29.848 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:29.849 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:29.849 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:29.849 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:29.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.846+0000 7f61527e3640 1 Processor -- start 2024-03-29T11:38:29.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.847+0000 7f61527e3640 1 -- start start 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.847+0000 7f61527e3640 10 monclient: init 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 10 monclient(hunting): picked mon.noname-c con 0x7f6144034850 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 10 monclient(hunting): picked mon.noname-a con 0x7f614402a640 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 10 monclient(hunting): picked mon.noname-b con 0x7f6144006410 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614402c820 con 0x7f614402a640 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614402a120 con 0x7f6144034850 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61440c75b0 con 0x7f6144006410 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:29.852 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.848+0000 7f61527e3640 10 monclient(hunting): authenticate will time out at 1066.113601s 2024-03-29T11:38:29.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.853+0000 7f6151fe2640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f6144006410 0x7f61440cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:60459/0 (socket says 172.21.15.33:60459) 2024-03-29T11:38:29.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.853+0000 7f6151fe2640 1 -- 172.21.15.33:0/3649055437 learned_addr learned my addr 172.21.15.33:0/3649055437 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:29.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.853+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234668052 0 0) 0x7f61440c75b0 con 0x7f6144006410 2024-03-29T11:38:29.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6138003520 con 0x7f6144006410 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030904831 0 0) 0x7f614402a120 con 0x7f6144034850 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61440c75b0 con 0x7f6144034850 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544845293 0 0) 0x7f6138003520 con 0x7f6144006410 2024-03-29T11:38:29.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 >> v1:172.21.15.153:6789/0 conn(0x7f6144034850 legacy=0x7f61440c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 >> v1:172.21.15.33:6789/0 conn(0x7f614402a640 legacy=0x7f61440320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 monclient: found mon.noname-b 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144033110 con 0x7f6144006410 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f6148001690 con 0x7f6144006410 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:29.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.854+0000 7f6143fff640 10 monclient: dump: 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61480017f0 con 0x7f6144006410 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f6143fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f6143fff640 1 -- 172.21.15.33:0/3649055437 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f6148001c00 con 0x7f6144006410 2024-03-29T11:38:29.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f6143fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f6143fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f6143fff640 10 monclient: dump: 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:29.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f61527e3640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f61527e3640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f61527e3640 4 set_mon_vals no callback set 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f61527e3640 10 monclient: shutdown 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f61527e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.855+0000 7f61527e3640 1 -- 172.21.15.33:0/3649055437 >> v1:172.21.15.33:6790/0 conn(0x7f6144006410 legacy=0x7f61440cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.856+0000 7f61527e3640 1 -- 172.21.15.33:0/3649055437 shutdown_connections 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.856+0000 7f61527e3640 1 -- 172.21.15.33:0/3649055437 >> 172.21.15.33:0/3649055437 conn(0x7f61440058f0 msgr2=0x7f6144005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.856+0000 7f61527e3640 1 -- 172.21.15.33:0/3649055437 shutdown_connections 2024-03-29T11:38:29.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.856+0000 7f61527e3640 1 -- 172.21.15.33:0/3649055437 wait complete. 2024-03-29T11:38:29.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.857+0000 7f61527e3640 10 monclient: build_initial_monmap 2024-03-29T11:38:29.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.857+0000 7f61527e3640 10 monclient: monmap: 2024-03-29T11:38:29.863 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:29.863 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:29.863 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:29.863 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:29.863 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.857+0000 7f61527e3640 1 librados: starting msgr at 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.857+0000 7f61527e3640 1 librados: starting objecter 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.858+0000 7f61527e3640 1 Processor -- start 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.858+0000 7f61527e3640 1 -- start start 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.858+0000 7f61527e3640 1 librados: setting wanted keys 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.858+0000 7f61527e3640 1 librados: calling monclient init 2024-03-29T11:38:29.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.858+0000 7f61527e3640 10 monclient: init 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.858+0000 7f61527e3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.858+0000 7f61527e3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 10 monclient(hunting): picked mon.noname-b con 0x7f6144006410 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 10 monclient(hunting): picked mon.noname-a con 0x7f614402a640 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 10 monclient(hunting): picked mon.noname-c con 0x7f6144034850 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614415bdb0 con 0x7f614402a640 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61440bc960 con 0x7f6144006410 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614402d9d0 con 0x7f6144034850 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:29.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.860+0000 7f61527e3640 10 monclient(hunting): authenticate will time out at 1066.125413s 2024-03-29T11:38:29.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.862+0000 7f61517e1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6144006410 0x7f6144158300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:39277/0 (socket says 172.21.15.33:39277) 2024-03-29T11:38:29.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.862+0000 7f61517e1640 1 -- 172.21.15.33:0/1017706019 learned_addr learned my addr 172.21.15.33:0/1017706019 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.863+0000 7f6141ffb640 10 client.?.objecter ms_handle_connect 0x7f6144006410 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.863+0000 7f6141ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.863+0000 7f6141ffb640 10 client.?.objecter ms_handle_connect 0x7f6144034850 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.863+0000 7f6141ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231525470 0 0) 0x7f61440bc960 con 0x7f6144006410 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6130004eb0 con 0x7f6144006410 2024-03-29T11:38:29.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1107628167 0 0) 0x7f6130004eb0 con 0x7f6144006410 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6130005300 con 0x7f6144006410 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306596149 0 0) 0x7f614402d9d0 con 0x7f6144034850 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.864+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6130005760 con 0x7f6144034850 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f613c0018d0 con 0x7f6144006410 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:29.874 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964976906 0 0) 0x7f6130005760 con 0x7f6144034850 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f614402d9d0 con 0x7f6144034850 2024-03-29T11:38:29.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f6148001b00 con 0x7f6144034850 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:29.876 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 993866593 0 0) 0x7f6130005300 con 0x7f6144006410 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 >> v1:172.21.15.33:6790/0 conn(0x7f6144034850 legacy=0x7f61440bc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.865+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 >> v1:172.21.15.33:6789/0 conn(0x7f614402a640 legacy=0x7f61440bbb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 1 monclient: found mon.b 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614415b800 con 0x7f6144006410 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f61527e3640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:38:29.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f61527e3640 10 monclient: _renew_subs 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f61527e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614415ba50 con 0x7f6144006410 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f613c0035f0 con 0x7f6144006410 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f613c0018d0 con 0x7f6144006410 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.866+0000 7f6141ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f6141ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f6141ffb640 10 monclient: dump: 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.878 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f61527e3640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f61527e3640 10 monclient: _renew_subs 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f61527e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6144031410 con 0x7f6144006410 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f61527e3640 1 librados: init done 2024-03-29T11:38:29.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.867+0000 7f61437fe640 4 set_mon_vals no callback set 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.869+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f613c003860 con 0x7f6144006410 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f613c0845d0 con 0x7f6144006410 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f6141ffb640 10 client.4366.objecter ms_dispatch 0x7f614c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f6141ffb640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f6141ffb640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f6141ffb640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f61527e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f61527e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f61527e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.870+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6144004cd0 con 0x7f6144006410 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.871+0000 7f6150fe0640 10 monclient: get_auth_request con 0x7f613006e5f0 auth_method 0 2024-03-29T11:38:29.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.871+0000 7f6141ffb640 10 client.4366.objecter ms_handle_connect 0x7f613006e5f0 2024-03-29T11:38:29.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.879+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f613c04fce0 con 0x7f6144006410 2024-03-29T11:38:29.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.880+0000 7f6141ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:29.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:29.880+0000 7f6141ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:30.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.213+0000 7f61527e3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:30.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.213+0000 7f61527e3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:30.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.213+0000 7f61527e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:30.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.213+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61440086f0 con 0x7f6144006410 2024-03-29T11:38:30.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.216+0000 7f6141ffb640 1 -- 172.21.15.33:0/1017706019 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f613c001730 con 0x7f6144006410 2024-03-29T11:38:30.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.216+0000 7f6141ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:30.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.216+0000 7f6141ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:30.218 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:30.218 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:30.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.222+0000 7f61527e3640 10 librados: watch_flush enter 2024-03-29T11:38:30.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 10 librados: watch_flush exit 2024-03-29T11:38:30.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:38:30.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:38:30.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 >> v1:172.21.15.153:6816/2737028074 conn(0x7f613006e5f0 legacy=0x7f6130070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 10 monclient: shutdown 2024-03-29T11:38:30.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:30.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 >> v1:172.21.15.153:6789/0 conn(0x7f6144006410 legacy=0x7f6144158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:30.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 shutdown_connections 2024-03-29T11:38:30.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.224+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 >> 172.21.15.33:0/1017706019 conn(0x7f61440058f0 msgr2=0x7f61440043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:30.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.225+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 shutdown_connections 2024-03-29T11:38:30.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.225+0000 7f61527e3640 1 -- 172.21.15.33:0/1017706019 wait complete. 2024-03-29T11:38:30.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:30.225+0000 7f61527e3640 1 librados: shutdown 2024-03-29T11:38:30.243 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:37.051 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:37.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.207+0000 7f90df59e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.207+0000 7f90df59e640 10 monclient: build_initial_monmap 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.207+0000 7f90df59e640 10 monclient: monmap: 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:37.208126+0000 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:37.208126+0000 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:37.208 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.209+0000 7f90df59e640 1 Processor -- start 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.209+0000 7f90df59e640 1 -- start start 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.209+0000 7f90df59e640 10 monclient: init 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 10 monclient(hunting): picked mon.noname-c con 0x7f90e00b4920 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 10 monclient(hunting): picked mon.noname-b con 0x7f90e00b4200 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 10 monclient(hunting): picked mon.noname-a con 0x7f90e013f9a0 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e00b0930 con 0x7f90e013f9a0 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e00af460 con 0x7f90e00b4920 2024-03-29T11:38:37.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e00aeda0 con 0x7f90e00b4200 2024-03-29T11:38:37.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:37.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.210+0000 7f90df59e640 10 monclient(hunting): authenticate will time out at 1073.474333s 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90ddd9b640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f90e00b4200 0x7f90e00b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:56933/0 (socket says 172.21.15.33:56933) 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90ddd9b640 1 -- 172.21.15.33:0/351566248 learned_addr learned my addr 172.21.15.33:0/351566248 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344104479 0 0) 0x7f90e00aeda0 con 0x7f90e00b4200 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c8003520 con 0x7f90e00b4200 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285889672 0 0) 0x7f90c8003520 con 0x7f90e00b4200 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 >> v1:172.21.15.153:6789/0 conn(0x7f90e00b4920 legacy=0x7f90e013f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:37.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.214+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 >> v1:172.21.15.33:6789/0 conn(0x7f90e013f9a0 legacy=0x7f90e014d8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:37.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:37.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 1 monclient: found mon.noname-b 2024-03-29T11:38:37.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:37.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:38:37.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e00af980 con 0x7f90e00b4200 2024-03-29T11:38:37.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 10 monclient: _finish_auth 0 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90df59e640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90df59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f90d8001a10 con 0x7f90e00b4200 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90dd59a640 10 monclient: dump: 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:37.219 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.215+0000 7f90df59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.216+0000 7f90dd59a640 1 -- 172.21.15.33:0/351566248 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90d8001c70 con 0x7f90e00b4200 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.216+0000 7f90dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.216+0000 7f90df59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.216+0000 7f90df59e640 4 set_mon_vals no callback set 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.217+0000 7f90df59e640 10 monclient: shutdown 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.217+0000 7f90df59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.217+0000 7f90df59e640 1 -- 172.21.15.33:0/351566248 >> v1:172.21.15.33:6790/0 conn(0x7f90e00b4200 legacy=0x7f90e00b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.218+0000 7f90df59e640 1 -- 172.21.15.33:0/351566248 shutdown_connections 2024-03-29T11:38:37.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.218+0000 7f90df59e640 1 -- 172.21.15.33:0/351566248 >> 172.21.15.33:0/351566248 conn(0x7f90e006d230 msgr2=0x7f90e0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:37.221 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.219+0000 7f90df59e640 1 -- 172.21.15.33:0/351566248 shutdown_connections 2024-03-29T11:38:37.221 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.219+0000 7f90df59e640 1 -- 172.21.15.33:0/351566248 wait complete. 2024-03-29T11:38:37.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.220+0000 7f90df59e640 10 monclient: build_initial_monmap 2024-03-29T11:38:37.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.220+0000 7f90df59e640 10 monclient: monmap: 2024-03-29T11:38:37.222 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:37.222 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:37.222 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:37.222 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.220+0000 7f90df59e640 1 librados: starting msgr at 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.220+0000 7f90df59e640 1 librados: starting objecter 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 1 Processor -- start 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 1 -- start start 2024-03-29T11:38:37.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 1 librados: setting wanted keys 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 1 librados: calling monclient init 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 10 monclient: init 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.221+0000 7f90df59e640 10 monclient(hunting): picked mon.noname-b con 0x7f90e00b4200 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.222+0000 7f90df59e640 10 monclient(hunting): picked mon.noname-a con 0x7f90e00b4920 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.222+0000 7f90df59e640 10 monclient(hunting): picked mon.noname-c con 0x7f90e013f9a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.222+0000 7f90df59e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e00aae30 con 0x7f90e00b4920 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.222+0000 7f90df59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e00abe30 con 0x7f90e00b4200 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.222+0000 7f90df59e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e00a94e0 con 0x7f90e013f9a0 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.222+0000 7f90df59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:37.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.222+0000 7f90df59e640 10 monclient(hunting): authenticate will time out at 1073.485723s 2024-03-29T11:38:37.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.226+0000 7f90ddd9b640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f90e00b4920 0x7f90e00ac530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:47087/0 (socket says 172.21.15.33:47087) 2024-03-29T11:38:37.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.226+0000 7f90ddd9b640 1 -- 172.21.15.33:0/3252287644 learned_addr learned my addr 172.21.15.33:0/3252287644 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 10 client.?.objecter ms_handle_connect 0x7f90e00b4920 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093277357 0 0) 0x7f90e00aae30 con 0x7f90e00b4920 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:38:37.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.227+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c0004eb0 con 0x7f90e00b4920 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358430282 0 0) 0x7f90c0004eb0 con 0x7f90e00b4920 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90e00aae30 con 0x7f90e00b4920 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f90d80019d0 con 0x7f90e00b4920 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): dump: 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:37.230 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 client.?.objecter ms_handle_connect 0x7f90e013f9a0 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776585061 0 0) 0x7f90e00a94e0 con 0x7f90e013f9a0 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:37.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.229+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c0004eb0 con 0x7f90e013f9a0 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 client.?.objecter ms_handle_connect 0x7f90e00b4200 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119145774 0 0) 0x7f90c0004eb0 con 0x7f90e013f9a0 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90e00a94e0 con 0x7f90e013f9a0 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1854233290 0 0) 0x7f90e00aae30 con 0x7f90e00b4920 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 >> v1:172.21.15.33:6790/0 conn(0x7f90e013f9a0 legacy=0x7f90e01ff880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 >> v1:172.21.15.153:6789/0 conn(0x7f90e00b4200 legacy=0x7f90e0071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 monclient: found mon.a 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:37.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e00a9670 con 0x7f90e00b4920 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 10 monclient: _renew_subs 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e00a9ab0 con 0x7f90e00b4920 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 10 monclient: _renew_subs 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e00aa910 con 0x7f90e00b4920 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90df59e640 1 librados: init done 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90d8005350 con 0x7f90e00b4920 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f90d8006690 con 0x7f90e00b4920 2024-03-29T11:38:37.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.230+0000 7f90ceffd640 10 monclient: dump: 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.231+0000 7f90cffff640 4 set_mon_vals no callback set 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f90d8001f20 con 0x7f90e00b4920 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90df59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:37.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f90d8085170 con 0x7f90e00b4920 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90df59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90ceffd640 10 client.4362.objecter ms_dispatch 0x7f90e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90ceffd640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90ceffd640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90df59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c4005080 con 0x7f90e00b4920 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.233+0000 7f90ceffd640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.234+0000 7f90de59c640 10 monclient: get_auth_request con 0x7f90c006dd70 auth_method 0 2024-03-29T11:38:37.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.234+0000 7f90ceffd640 10 client.4362.objecter ms_handle_connect 0x7f90c006dd70 2024-03-29T11:38:37.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.245+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90d8050880 con 0x7f90e00b4920 2024-03-29T11:38:37.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.245+0000 7f90ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:37.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.245+0000 7f90ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:37.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.574+0000 7f90df59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:37.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.574+0000 7f90df59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:37.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.574+0000 7f90df59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:37.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.574+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90c4005330 con 0x7f90e00b4920 2024-03-29T11:38:37.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.578+0000 7f90ceffd640 1 -- 172.21.15.33:0/3252287644 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f90d8052980 con 0x7f90e00b4920 2024-03-29T11:38:37.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.578+0000 7f90ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:37.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.578+0000 7f90ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:37.580 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:37.580 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:37.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.583+0000 7f90df59e640 10 librados: watch_flush enter 2024-03-29T11:38:37.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.583+0000 7f90df59e640 10 librados: watch_flush exit 2024-03-29T11:38:37.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.583+0000 7f90df59e640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:38:37.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.583+0000 7f90df59e640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:38:37.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.586+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 >> v1:172.21.15.153:6816/2737028074 conn(0x7f90c006dd70 legacy=0x7f90c0070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.586+0000 7f90df59e640 10 monclient: shutdown 2024-03-29T11:38:37.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.586+0000 7f90df59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:37.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.586+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 >> v1:172.21.15.33:6789/0 conn(0x7f90e00b4920 legacy=0x7f90e00ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:37.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.586+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 shutdown_connections 2024-03-29T11:38:37.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.586+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 >> 172.21.15.33:0/3252287644 conn(0x7f90e006d230 msgr2=0x7f90e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:37.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.587+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 shutdown_connections 2024-03-29T11:38:37.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.587+0000 7f90df59e640 1 -- 172.21.15.33:0/3252287644 wait complete. 2024-03-29T11:38:37.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:37.587+0000 7f90df59e640 1 librados: shutdown 2024-03-29T11:38:37.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:44.412 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.562+0000 7f36b747e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.562+0000 7f36b747e640 10 monclient: build_initial_monmap 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.562+0000 7f36b747e640 10 monclient: monmap: 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:44.563089+0000 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:44.563089+0000 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:44.563 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:44.564 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:44.564 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:44.564 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:44.564 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:44.564 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.563+0000 7f36b747e640 1 Processor -- start 2024-03-29T11:38:44.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.563+0000 7f36b747e640 1 -- start start 2024-03-29T11:38:44.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.563+0000 7f36b747e640 10 monclient: init 2024-03-29T11:38:44.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.564+0000 7f36b747e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:44.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.565+0000 7f36b747e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 10 monclient(hunting): picked mon.noname-b con 0x7f36b00a9750 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 10 monclient(hunting): picked mon.noname-a con 0x7f36b00aad20 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 10 monclient(hunting): picked mon.noname-c con 0x7f36b00a9b30 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b00af4d0 con 0x7f36b00aad20 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b00af7b0 con 0x7f36b00a9b30 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b00acab0 con 0x7f36b00a9750 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.567+0000 7f36b747e640 10 monclient(hunting): authenticate will time out at 1080.831658s 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b647c640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f36b00a9750 0x7f36b00ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:56043/0 (socket says 172.21.15.33:56043) 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b647c640 1 -- 172.21.15.33:0/1351038123 learned_addr learned my addr 172.21.15.33:0/1351038123 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:44.569 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877279260 0 0) 0x7f36b00acab0 con 0x7f36b00a9750 2024-03-29T11:38:44.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b547a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:44.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b547a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:44.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b547a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:44.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b547a640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:38:44.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.568+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c005730 con 0x7f36b00a9750 2024-03-29T11:38:44.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.571+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2482878093 0 0) 0x7f369c005730 con 0x7f36b00a9750 2024-03-29T11:38:44.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.571+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 >> v1:172.21.15.153:6789/0 conn(0x7f36b00a9b30 legacy=0x7f36b0149930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 >> v1:172.21.15.33:6789/0 conn(0x7f36b00aad20 legacy=0x7f36b0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 1 monclient: found mon.noname-b 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b00aba30 con 0x7f36b00a9750 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 10 monclient: _finish_auth 0 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f36a0001690 con 0x7f36b00a9750 2024-03-29T11:38:44.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b547a640 10 monclient: dump: 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:44.578 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b747e640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.574+0000 7f36b747e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a00017f0 con 0x7f36b00a9750 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b547a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b547a640 1 -- 172.21.15.33:0/1351038123 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f36a0001c00 con 0x7f36b00a9750 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b547a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b547a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b547a640 10 monclient: dump: 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:44.579 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:44.580 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:44.580 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:44.580 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:44.580 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b747e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:44.580 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.577+0000 7f36b747e640 4 set_mon_vals no callback set 2024-03-29T11:38:44.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.578+0000 7f36b747e640 10 monclient: shutdown 2024-03-29T11:38:44.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.578+0000 7f36b747e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:44.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.578+0000 7f36b747e640 1 -- 172.21.15.33:0/1351038123 >> v1:172.21.15.33:6790/0 conn(0x7f36b00a9750 legacy=0x7f36b00ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.579+0000 7f36b747e640 1 -- 172.21.15.33:0/1351038123 shutdown_connections 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.579+0000 7f36b747e640 1 -- 172.21.15.33:0/1351038123 >> 172.21.15.33:0/1351038123 conn(0x7f36b00673e0 msgr2=0x7f36b00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.579+0000 7f36b747e640 1 -- 172.21.15.33:0/1351038123 shutdown_connections 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.579+0000 7f36b747e640 1 -- 172.21.15.33:0/1351038123 wait complete. 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.580+0000 7f36b747e640 10 monclient: build_initial_monmap 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.580+0000 7f36b747e640 10 monclient: monmap: 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:44.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.581+0000 7f36b747e640 1 librados: starting msgr at 2024-03-29T11:38:44.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.581+0000 7f36b747e640 1 librados: starting objecter 2024-03-29T11:38:44.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.581+0000 7f36b747e640 1 Processor -- start 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.581+0000 7f36b747e640 1 -- start start 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.582+0000 7f36b747e640 1 librados: setting wanted keys 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.582+0000 7f36b747e640 1 librados: calling monclient init 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.582+0000 7f36b747e640 10 monclient: init 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.583+0000 7f36b747e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.583+0000 7f36b747e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.583+0000 7f36b747e640 10 monclient(hunting): picked mon.noname-c con 0x7f36b00a9750 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:44.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.583+0000 7f36b747e640 10 monclient(hunting): picked mon.noname-b con 0x7f36b00a9b30 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:44.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.583+0000 7f36b747e640 10 monclient(hunting): picked mon.noname-a con 0x7f36b00aad20 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:44.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.586+0000 7f36b747e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b0070da0 con 0x7f36b00aad20 2024-03-29T11:38:44.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.586+0000 7f36b747e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b00708a0 con 0x7f36b00a9b30 2024-03-29T11:38:44.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.586+0000 7f36b747e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b013c3a0 con 0x7f36b00a9750 2024-03-29T11:38:44.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.586+0000 7f36b747e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:44.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.586+0000 7f36b747e640 10 monclient(hunting): authenticate will time out at 1080.850544s 2024-03-29T11:38:44.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.590+0000 7f36b5c7b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f36b00a9b30 0x7f36b013a650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:48863/0 (socket says 172.21.15.33:48863) 2024-03-29T11:38:44.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.590+0000 7f36b5c7b640 1 -- 172.21.15.33:0/3588734475 learned_addr learned my addr 172.21.15.33:0/3588734475 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:44.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.590+0000 7f36a6ffd640 10 client.?.objecter ms_handle_connect 0x7f36b00aad20 2024-03-29T11:38:44.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.590+0000 7f36a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:44.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 10 client.?.objecter ms_handle_connect 0x7f36b00a9b30 2024-03-29T11:38:44.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005906706 0 0) 0x7f36b0070da0 con 0x7f36b00aad20 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.591+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3694004eb0 con 0x7f36b00aad20 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.592+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265960090 0 0) 0x7f3694004eb0 con 0x7f36b00aad20 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.592+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3694005300 con 0x7f36b00aad20 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.592+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f36ac0017b0 con 0x7f36b00aad20 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.592+0000 7f36a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.592+0000 7f36a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.592+0000 7f36a6ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.593 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.594 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.594 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:44.594 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:44.594 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:44.594 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:44.594 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:44.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.592+0000 7f36a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.593+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434871972 0 0) 0x7f36b00708a0 con 0x7f36b00a9b30 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.593+0000 7f36a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.593+0000 7f36a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.593+0000 7f36a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.593+0000 7f36a6ffd640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.593+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3694005850 con 0x7f36b00a9b30 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.594+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1729728475 0 0) 0x7f3694005850 con 0x7f36b00a9b30 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.594+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3694005ce0 con 0x7f36b00a9b30 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f36a8001b90 con 0x7f36b00a9b30 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:44.597 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 675670255 0 0) 0x7f3694005300 con 0x7f36b00aad20 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 >> v1:172.21.15.33:6790/0 conn(0x7f36b00a9750 legacy=0x7f36b006f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 >> v1:172.21.15.153:6789/0 conn(0x7f36b00a9b30 legacy=0x7f36b013a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 monclient: found mon.a 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:44.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0070450 con 0x7f36b00aad20 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 10 monclient: _renew_subs 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b006ff20 con 0x7f36b00aad20 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 10 monclient: _renew_subs 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b013bae0 con 0x7f36b00aad20 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36b747e640 1 librados: init done 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36ac001d40 con 0x7f36b00aad20 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f36ac0017b0 con 0x7f36b00aad20 2024-03-29T11:38:44.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.595+0000 7f36a6ffd640 10 monclient: dump: 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:44.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.596+0000 7f36b4c79640 4 set_mon_vals no callback set 2024-03-29T11:38:44.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.597+0000 7f36a4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.597+0000 7f36a4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.597+0000 7f36a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.597+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b0070270 con 0x7f36b00aad20 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.598+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f36ac019500 con 0x7f36b00aad20 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.599+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f36ac083310 con 0x7f36b00aad20 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.599+0000 7f36a6ffd640 10 client.4368.objecter ms_dispatch 0x7f36b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.599+0000 7f36a6ffd640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.599+0000 7f36a6ffd640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.599+0000 7f36a6ffd640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.599+0000 7f36b647c640 10 monclient: get_auth_request con 0x7f369406e940 auth_method 0 2024-03-29T11:38:44.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.600+0000 7f36a6ffd640 10 client.4368.objecter ms_handle_connect 0x7f369406e940 2024-03-29T11:38:44.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.605+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36ac04ea20 con 0x7f36b00aad20 2024-03-29T11:38:44.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.605+0000 7f36a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:44.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.605+0000 7f36a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:44.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.920+0000 7f36a4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:44.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.920+0000 7f36a4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:44.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.920+0000 7f36a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:38:44.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.920+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b0070a30 con 0x7f36b00aad20 2024-03-29T11:38:44.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.923+0000 7f36a6ffd640 1 -- 172.21.15.33:0/3588734475 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f36ac001610 con 0x7f36b00aad20 2024-03-29T11:38:44.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.923+0000 7f36a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:44.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.923+0000 7f36a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:44.926 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 10 librados: watch_flush enter 2024-03-29T11:38:44.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 10 librados: watch_flush exit 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 >> v1:172.21.15.153:6816/2737028074 conn(0x7f369406e940 legacy=0x7f3694070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 10 monclient: shutdown 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 >> v1:172.21.15.33:6789/0 conn(0x7f36b00aad20 legacy=0x7f36b013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 shutdown_connections 2024-03-29T11:38:44.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.928+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 >> 172.21.15.33:0/3588734475 conn(0x7f36b00673e0 msgr2=0x7f36b00b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.930+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 shutdown_connections 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.930+0000 7f36a4ff9640 1 -- 172.21.15.33:0/3588734475 wait complete. 2024-03-29T11:38:44.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:44.930+0000 7f36a4ff9640 1 librados: shutdown 2024-03-29T11:38:44.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:51.759 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:51.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.923+0000 7fcdf588f640 10 monclient: get_monmap_and_config 2024-03-29T11:38:51.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.923+0000 7fcdf588f640 10 monclient: build_initial_monmap 2024-03-29T11:38:51.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.923+0000 7fcdf588f640 10 monclient: monmap: 2024-03-29T11:38:51.924 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:51.924 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:51.924330+0000 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:51.924330+0000 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:51.925 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:51.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.924+0000 7fcdf588f640 1 Processor -- start 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.924+0000 7fcdf588f640 1 -- start start 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.924+0000 7fcdf588f640 10 monclient: init 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf01457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf013f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf00a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:51.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf00aaef0 con 0x7fcdf00a9d70 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf0145bd0 con 0x7fcdf01457f0 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf0145eb0 con 0x7fcdf013f650 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdf588f640 10 monclient(hunting): authenticate will time out at 1088.189441s 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.925+0000 7fcdef7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fcdf00a9d70 0x7fcdf014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:52356/0 (socket says 172.21.15.33:52356) 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.926+0000 7fcdef7fe640 1 -- 172.21.15.33:0/1631413827 learned_addr learned my addr 172.21.15.33:0/1631413827 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.926+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453474804 0 0) 0x7fcdf00aaef0 con 0x7fcdf00a9d70 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.926+0000 7fcdedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:51.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.926+0000 7fcdedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.926+0000 7fcdedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.926+0000 7fcdedffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.926+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd8003520 con 0x7fcdf00a9d70 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602419310 0 0) 0x7fcdf0145bd0 con 0x7fcdf01457f0 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf00aaef0 con 0x7fcdf01457f0 2024-03-29T11:38:51.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 546838408 0 0) 0x7fcdd8003520 con 0x7fcdf00a9d70 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 >> v1:172.21.15.33:6790/0 conn(0x7fcdf013f650 legacy=0x7fcdf00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 >> v1:172.21.15.153:6789/0 conn(0x7fcdf01457f0 legacy=0x7fcdf013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 monclient: found mon.noname-a 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0146340 con 0x7fcdf00a9d70 2024-03-29T11:38:51.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdf588f640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdf588f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fcde8001a10 con 0x7fcdf00a9d70 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.927+0000 7fcdedffb640 10 monclient: dump: 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:51.935 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdf588f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde8001c30 con 0x7fcdf00a9d70 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdedffb640 1 -- 172.21.15.33:0/1631413827 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fcde8001fc0 con 0x7fcdf00a9d70 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdedffb640 10 monclient: dump: 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:51.936 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdf588f640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.928+0000 7fcdf588f640 4 set_mon_vals no callback set 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.929+0000 7fcdf588f640 10 monclient: shutdown 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.929+0000 7fcdf588f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.929+0000 7fcdf588f640 1 -- 172.21.15.33:0/1631413827 >> v1:172.21.15.33:6789/0 conn(0x7fcdf00a9d70 legacy=0x7fcdf014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.929+0000 7fcdf588f640 1 -- 172.21.15.33:0/1631413827 shutdown_connections 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.929+0000 7fcdf588f640 1 -- 172.21.15.33:0/1631413827 >> 172.21.15.33:0/1631413827 conn(0x7fcdf0066a00 msgr2=0x7fcdf0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.929+0000 7fcdf588f640 1 -- 172.21.15.33:0/1631413827 shutdown_connections 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.929+0000 7fcdf588f640 1 -- 172.21.15.33:0/1631413827 wait complete. 2024-03-29T11:38:51.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.930+0000 7fcdf588f640 10 monclient: build_initial_monmap 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.930+0000 7fcdf588f640 10 monclient: monmap: 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:51.938 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.931+0000 7fcdf588f640 1 librados: starting msgr at 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.931+0000 7fcdf588f640 1 librados: starting objecter 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.931+0000 7fcdf588f640 1 Processor -- start 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.932+0000 7fcdf588f640 1 -- start start 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.932+0000 7fcdf588f640 1 librados: setting wanted keys 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.932+0000 7fcdf588f640 1 librados: calling monclient init 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.932+0000 7fcdf588f640 10 monclient: init 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:51.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf00a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf013f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf01457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf0137b70 con 0x7fcdf01457f0 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf0137e50 con 0x7fcdf013f650 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf0138130 con 0x7fcdf00a9d70 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdf588f640 10 monclient(hunting): authenticate will time out at 1088.198279s 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdeeffd640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fcdf00a9d70 0x7fcdf0138550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:49902/0 (socket says 172.21.15.33:49902) 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdeeffd640 1 -- 172.21.15.33:0/4062823086 learned_addr learned my addr 172.21.15.33:0/4062823086 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdd7fff640 10 client.?.objecter ms_handle_connect 0x7fcdf00a9d70 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.934+0000 7fcdd7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 client.?.objecter ms_handle_connect 0x7fcdf01457f0 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 client.?.objecter ms_handle_connect 0x7fcdf013f650 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227284773 0 0) 0x7fcdf0137e50 con 0x7fcdf013f650 2024-03-29T11:38:51.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.935+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcddc004eb0 con 0x7fcdf013f650 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588857600 0 0) 0x7fcdf0138130 con 0x7fcdf00a9d70 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf0137e50 con 0x7fcdf00a9d70 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490838948 0 0) 0x7fcdf0137b70 con 0x7fcdf01457f0 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf0138130 con 0x7fcdf01457f0 2024-03-29T11:38:51.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.936+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4148724128 0 0) 0x7fcddc004eb0 con 0x7fcdf013f650 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdf0137b70 con 0x7fcdf013f650 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fcde4001930 con 0x7fcdf013f650 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient(hunting): dump: 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:51.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 308022631 0 0) 0x7fcdf0137b70 con 0x7fcdf013f650 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 >> v1:172.21.15.33:6790/0 conn(0x7fcdf00a9d70 legacy=0x7fcdf0138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 >> v1:172.21.15.33:6789/0 conn(0x7fcdf01457f0 legacy=0x7fcdf01e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 1 monclient: found mon.b 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0136f10 con 0x7fcdf013f650 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.937+0000 7fcdd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 10 monclient: _renew_subs 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf0138c60 con 0x7fcdf013f650 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:51.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 10 monclient: _renew_subs 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf0135d60 con 0x7fcdf013f650 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde4001a90 con 0x7fcdf013f650 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdf588f640 1 librados: init done 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcded7fa640 4 set_mon_vals no callback set 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fcde4003830 con 0x7fcdf013f650 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.938+0000 7fcdd7fff640 10 monclient: dump: 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:51.947 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.940+0000 7fcdf588f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.940+0000 7fcdf588f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.940+0000 7fcdf588f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.940+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf00b6ea0 con 0x7fcdf013f650 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.940+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fcde4003990 con 0x7fcdf013f650 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.941+0000 7fcdeeffd640 10 monclient: get_auth_request con 0x7fcddc06dc90 auth_method 0 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.941+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fcde40838b0 con 0x7fcdf013f650 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.941+0000 7fcdd7fff640 10 client.4378.objecter ms_dispatch 0x7fcdf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.941+0000 7fcdd7fff640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.941+0000 7fcdd7fff640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.941+0000 7fcdd7fff640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T11:38:51.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.941+0000 7fcdd7fff640 10 client.4378.objecter ms_handle_connect 0x7fcddc06dc90 2024-03-29T11:38:51.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.947+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcde404efc0 con 0x7fcdf013f650 2024-03-29T11:38:51.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.947+0000 7fcdd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:51.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:51.947+0000 7fcdd7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:52.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.433+0000 7fcdf588f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:52.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.433+0000 7fcdf588f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:52.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.433+0000 7fcdf588f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:52.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.433+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdf0075f10 con 0x7fcdf013f650 2024-03-29T11:38:52.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.433+0000 7fcdd7fff640 1 -- 172.21.15.33:0/4062823086 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fcde40510c0 con 0x7fcdf013f650 2024-03-29T11:38:52.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.434+0000 7fcdd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:52.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.434+0000 7fcdd7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:52.435 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:52.435 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:52.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 10 librados: watch_flush enter 2024-03-29T11:38:52.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 10 librados: watch_flush exit 2024-03-29T11:38:52.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T11:38:52.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 10 client.4378.objecter successfully canceled tick 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 >> v1:172.21.15.153:6816/2737028074 conn(0x7fcddc06dc90 legacy=0x7fcddc070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 10 monclient: shutdown 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 >> v1:172.21.15.153:6789/0 conn(0x7fcdf013f650 legacy=0x7fcdf01415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.437+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 shutdown_connections 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.438+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 >> 172.21.15.33:0/4062823086 conn(0x7fcdf0066a00 msgr2=0x7fcdf00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.438+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 shutdown_connections 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.438+0000 7fcdf588f640 1 -- 172.21.15.33:0/4062823086 wait complete. 2024-03-29T11:38:52.439 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:52.438+0000 7fcdf588f640 1 librados: shutdown 2024-03-29T11:38:52.452 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:59.259 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.445+0000 7fb1ad59d640 10 monclient: get_monmap_and_config 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.445+0000 7fb1ad59d640 10 monclient: build_initial_monmap 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.445+0000 7fb1ad59d640 10 monclient: monmap: 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:38:59.445769+0000 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:38:59.445769+0000 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:38:59.446 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:59.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.446+0000 7fb1ad59d640 1 Processor -- start 2024-03-29T11:38:59.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.446+0000 7fb1ad59d640 1 -- start start 2024-03-29T11:38:59.447 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.446+0000 7fb1ad59d640 10 monclient: init 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a81457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a813f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a80a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a80aaef0 con 0x7fb1a81457f0 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a8145bd0 con 0x7fb1a813f650 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a8145eb0 con 0x7fb1a80a9d70 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1ad59d640 10 monclient(hunting): authenticate will time out at 1095.711211s 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1a6ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fb1a81457f0 0x7fb1a813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:56000/0 (socket says 172.21.15.33:56000) 2024-03-29T11:38:59.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.447+0000 7fb1a6ffd640 1 -- 172.21.15.33:0/3468016321 learned_addr learned my addr 172.21.15.33:0/3468016321 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:59.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873072586 0 0) 0x7fb1a8145bd0 con 0x7fb1a813f650 2024-03-29T11:38:59.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:38:59.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190003520 con 0x7fb1a813f650 2024-03-29T11:38:59.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113241224 0 0) 0x7fb1a80aaef0 con 0x7fb1a81457f0 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.448+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a8145bd0 con 0x7fb1a81457f0 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2024068403 0 0) 0x7fb1a8145bd0 con 0x7fb1a81457f0 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 >> v1:172.21.15.33:6790/0 conn(0x7fb1a80a9d70 legacy=0x7fb1a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 >> v1:172.21.15.153:6789/0 conn(0x7fb1a813f650 legacy=0x7fb1a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a8146340 con 0x7fb1a81457f0 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:59.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb19c001670 con 0x7fb1a81457f0 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: dump: 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1ad59d640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:38:59.451 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1ad59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb19c0017d0 con 0x7fb1a81457f0 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1ad59d640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 1 -- 172.21.15.33:0/3468016321 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb19c001c20 con 0x7fb1a81457f0 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1ad59d640 4 set_mon_vals no callback set 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.449+0000 7fb1a5ffb640 10 monclient: dump: 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.453 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.450+0000 7fb1ad59d640 10 monclient: shutdown 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.450+0000 7fb1ad59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.450+0000 7fb1ad59d640 1 -- 172.21.15.33:0/3468016321 >> v1:172.21.15.33:6789/0 conn(0x7fb1a81457f0 legacy=0x7fb1a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.450+0000 7fb1ad59d640 1 -- 172.21.15.33:0/3468016321 shutdown_connections 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.450+0000 7fb1ad59d640 1 -- 172.21.15.33:0/3468016321 >> 172.21.15.33:0/3468016321 conn(0x7fb1a8066a00 msgr2=0x7fb1a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.450+0000 7fb1ad59d640 1 -- 172.21.15.33:0/3468016321 shutdown_connections 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.450+0000 7fb1ad59d640 1 -- 172.21.15.33:0/3468016321 wait complete. 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.451+0000 7fb1ad59d640 10 monclient: build_initial_monmap 2024-03-29T11:38:59.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.451+0000 7fb1ad59d640 10 monclient: monmap: 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.452+0000 7fb1ad59d640 1 librados: starting msgr at 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.452+0000 7fb1ad59d640 1 librados: starting objecter 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.452+0000 7fb1ad59d640 1 Processor -- start 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.452+0000 7fb1ad59d640 1 -- start start 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.452+0000 7fb1ad59d640 1 librados: setting wanted keys 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.452+0000 7fb1ad59d640 1 librados: calling monclient init 2024-03-29T11:38:59.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.452+0000 7fb1ad59d640 10 monclient: init 2024-03-29T11:38:59.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:59.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:59.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a80a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:38:59.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a813f650 addr v1:172.21.15.33:6789/0 2024-03-29T11:38:59.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a81457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:38:59.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a8142d40 con 0x7fb1a813f650 2024-03-29T11:38:59.458 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a813bad0 con 0x7fb1a81457f0 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a813a950 con 0x7fb1a80a9d70 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.453+0000 7fb1ad59d640 10 monclient(hunting): authenticate will time out at 1095.717487s 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.454+0000 7fb1a77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb1a81457f0 0x7fb1a81e5ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:39086/0 (socket says 172.21.15.33:39086) 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.454+0000 7fb1a77fe640 1 -- 172.21.15.33:0/1954454939 learned_addr learned my addr 172.21.15.33:0/1954454939 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.454+0000 7fb18f7fe640 10 client.?.objecter ms_handle_connect 0x7fb1a80a9d70 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.454+0000 7fb18f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.454+0000 7fb18f7fe640 10 client.?.objecter ms_handle_connect 0x7fb1a81457f0 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.454+0000 7fb18f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810465780 0 0) 0x7fb1a813bad0 con 0x7fb1a81457f0 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb194004eb0 con 0x7fb1a81457f0 2024-03-29T11:38:59.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 client.?.objecter ms_handle_connect 0x7fb1a813f650 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604956640 0 0) 0x7fb1a8142d40 con 0x7fb1a813f650 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a813bad0 con 0x7fb1a813f650 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2571019722 0 0) 0x7fb194004eb0 con 0x7fb1a81457f0 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1a8142d40 con 0x7fb1a81457f0 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb1a0001c60 con 0x7fb1a81457f0 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.455+0000 7fb18f7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:59.460 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157316606 0 0) 0x7fb1a813a950 con 0x7fb1a80a9d70 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb194004eb0 con 0x7fb1a80a9d70 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3557472889 0 0) 0x7fb1a8142d40 con 0x7fb1a81457f0 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 >> v1:172.21.15.33:6790/0 conn(0x7fb1a80a9d70 legacy=0x7fb1a813b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.461 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 >> v1:172.21.15.33:6789/0 conn(0x7fb1a813f650 legacy=0x7fb1a8138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 1 monclient: found mon.b 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a813ac30 con 0x7fb1a81457f0 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb18f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 10 monclient: _renew_subs 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a8135930 con 0x7fb1a81457f0 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 10 monclient: _renew_subs 2024-03-29T11:38:59.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a81394f0 con 0x7fb1a81457f0 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.456+0000 7fb1ad59d640 1 librados: init done 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a0003990 con 0x7fb1a81457f0 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb18f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb1a0003ca0 con 0x7fb1a81457f0 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb18f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb18f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb1a57fa640 4 set_mon_vals no callback set 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb18f7fe640 10 monclient: dump: 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:38:59.463 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:38:59.464 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:38:59.464 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:38:59.464 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:38:59.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.458+0000 7fb1ad59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.459+0000 7fb1ad59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.459+0000 7fb1ad59d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.459+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb188005080 con 0x7fb1a81457f0 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.459+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fb1a0003ec0 con 0x7fb1a81457f0 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.460+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fb1a0084d90 con 0x7fb1a81457f0 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.460+0000 7fb18f7fe640 10 client.4384.objecter ms_dispatch 0x7fb1a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.460+0000 7fb18f7fe640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.460+0000 7fb18f7fe640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.460+0000 7fb18f7fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.460+0000 7fb1a6ffd640 10 monclient: get_auth_request con 0x7fb19406db70 auth_method 0 2024-03-29T11:38:59.465 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.460+0000 7fb18f7fe640 10 client.4384.objecter ms_handle_connect 0x7fb19406db70 2024-03-29T11:38:59.466 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.466+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1a00506f0 con 0x7fb1a81457f0 2024-03-29T11:38:59.466 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.466+0000 7fb18f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:59.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.466+0000 7fb18f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:59.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.887+0000 7fb1ad59d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.887+0000 7fb1ad59d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.887+0000 7fb1ad59d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:38:59.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.887+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb188005330 con 0x7fb1a81457f0 2024-03-29T11:38:59.889 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:38:59.889 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:59.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.888+0000 7fb18f7fe640 1 -- 172.21.15.33:0/1954454939 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fb1a0084440 con 0x7fb1a81457f0 2024-03-29T11:38:59.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.888+0000 7fb18f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.888+0000 7fb18f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:59.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.892+0000 7fb1ad59d640 10 librados: watch_flush enter 2024-03-29T11:38:59.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.892+0000 7fb1ad59d640 10 librados: watch_flush exit 2024-03-29T11:38:59.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.892+0000 7fb1ad59d640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:38:59.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.892+0000 7fb1ad59d640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:38:59.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 >> v1:172.21.15.153:6816/2737028074 conn(0x7fb19406db70 legacy=0x7fb194070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 10 monclient: shutdown 2024-03-29T11:38:59.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:59.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 >> v1:172.21.15.153:6789/0 conn(0x7fb1a81457f0 legacy=0x7fb1a81e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 shutdown_connections 2024-03-29T11:38:59.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 >> 172.21.15.33:0/1954454939 conn(0x7fb1a8066a00 msgr2=0x7fb1a80b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:59.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 shutdown_connections 2024-03-29T11:38:59.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 1 -- 172.21.15.33:0/1954454939 wait complete. 2024-03-29T11:38:59.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:38:59.893+0000 7fb1ad59d640 1 librados: shutdown 2024-03-29T11:38:59.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:06.718 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:06.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.862+0000 7f0711b9d640 10 monclient: get_monmap_and_config 2024-03-29T11:39:06.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.862+0000 7f0711b9d640 10 monclient: build_initial_monmap 2024-03-29T11:39:06.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.862+0000 7f0711b9d640 10 monclient: monmap: 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:06.864525+0000 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:06.864525+0000 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:06.865 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:06.866 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.864+0000 7f0711b9d640 1 Processor -- start 2024-03-29T11:39:06.866 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.864+0000 7f0711b9d640 1 -- start start 2024-03-29T11:39:06.866 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.864+0000 7f0711b9d640 10 monclient: init 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.865+0000 7f0711b9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.865+0000 7f0711b9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.865+0000 7f0711b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0704006a70 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.865+0000 7f0711b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f070402b100 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f0711b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0704029c30 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f0711b9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070402a270 con 0x7f0704006a70 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f0711b9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070402c440 con 0x7f070402b100 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f0711b9d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040c9440 con 0x7f0704029c30 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f0711b9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:06.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f0711b9d640 10 monclient(hunting): authenticate will time out at 1103.130773s 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f070bfff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f0704029c30 0x7f07040d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:58868/0 (socket says 172.21.15.33:58868) 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.866+0000 7f070bfff640 1 -- 172.21.15.33:0/1312751039 learned_addr learned my addr 172.21.15.33:0/1312751039 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713180893 0 0) 0x7f07040c9440 con 0x7f0704029c30 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:39:06.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06fc003520 con 0x7f0704029c30 2024-03-29T11:39:06.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3402256774 0 0) 0x7f06fc003520 con 0x7f0704029c30 2024-03-29T11:39:06.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.867+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 >> v1:172.21.15.153:6789/0 conn(0x7f070402b100 legacy=0x7f07040cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:06.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 >> v1:172.21.15.33:6789/0 conn(0x7f0704006a70 legacy=0x7f07040c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:06.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:39:06.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:06.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:39:06.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040c95d0 con 0x7f0704029c30 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.869+0000 7f070a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f070c04f730 con 0x7f0704029c30 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 10 monclient: dump: 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:06.873 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f0711b9d640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f0711b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f070c075f10 con 0x7f0704029c30 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 1 -- 172.21.15.33:0/1312751039 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f070c0758b0 con 0x7f0704029c30 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f070a7fc640 10 monclient: dump: 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.874 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:06.875 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:06.875 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:06.875 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:06.875 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:06.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f0711b9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:06.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.870+0000 7f0711b9d640 4 set_mon_vals no callback set 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.871+0000 7f0711b9d640 10 monclient: shutdown 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.871+0000 7f0711b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.871+0000 7f0711b9d640 1 -- 172.21.15.33:0/1312751039 >> v1:172.21.15.33:6790/0 conn(0x7f0704029c30 legacy=0x7f07040d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.872+0000 7f0711b9d640 1 -- 172.21.15.33:0/1312751039 shutdown_connections 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.872+0000 7f0711b9d640 1 -- 172.21.15.33:0/1312751039 >> 172.21.15.33:0/1312751039 conn(0x7f07040058f0 msgr2=0x7f0704005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.872+0000 7f0711b9d640 1 -- 172.21.15.33:0/1312751039 shutdown_connections 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.873+0000 7f0711b9d640 1 -- 172.21.15.33:0/1312751039 wait complete. 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.874+0000 7f0711b9d640 10 monclient: build_initial_monmap 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.874+0000 7f0711b9d640 10 monclient: monmap: 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:06.876 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.874+0000 7f0711b9d640 1 librados: starting msgr at 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.874+0000 7f0711b9d640 1 librados: starting objecter 2024-03-29T11:39:06.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.874+0000 7f0711b9d640 1 Processor -- start 2024-03-29T11:39:06.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.875+0000 7f0711b9d640 1 -- start start 2024-03-29T11:39:06.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.875+0000 7f0711b9d640 1 librados: setting wanted keys 2024-03-29T11:39:06.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.875+0000 7f0711b9d640 1 librados: calling monclient init 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.875+0000 7f0711b9d640 10 monclient: init 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.875+0000 7f0711b9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.875+0000 7f0711b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.875+0000 7f0711b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0704006a70 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.876+0000 7f0711b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0704029c30 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.877+0000 7f0711b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0704161170 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.877+0000 7f0711b9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070402f5a0 con 0x7f0704161170 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.877+0000 7f0711b9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070402d810 con 0x7f0704006a70 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.877+0000 7f0711b9d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0704160c50 con 0x7f0704029c30 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.877+0000 7f0711b9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:06.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.877+0000 7f0711b9d640 10 monclient(hunting): authenticate will time out at 1103.142012s 2024-03-29T11:39:06.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.878+0000 7f070affd640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f0704029c30 0x7f070402d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:60618/0 (socket says 172.21.15.33:60618) 2024-03-29T11:39:06.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.878+0000 7f070affd640 1 -- 172.21.15.33:0/3116814394 learned_addr learned my addr 172.21.15.33:0/3116814394 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 10 client.?.objecter ms_handle_connect 0x7f0704006a70 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664140683 0 0) 0x7f070402d810 con 0x7f0704006a70 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:39:06.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.881+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f0004eb0 con 0x7f0704006a70 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 client.?.objecter ms_handle_connect 0x7f0704161170 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234664543 0 0) 0x7f070402f5a0 con 0x7f0704161170 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070402d810 con 0x7f0704161170 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 client.?.objecter ms_handle_connect 0x7f0704029c30 2024-03-29T11:39:06.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2326081147 0 0) 0x7f06f0004eb0 con 0x7f0704006a70 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f070402f5a0 con 0x7f0704006a70 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2792463728 0 0) 0x7f070402d810 con 0x7f0704161170 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f0004eb0 con 0x7f0704161170 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f06f80017c0 con 0x7f0704006a70 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:06.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): dump: 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f070c04f400 con 0x7f0704161170 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:06.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): dump: 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134027156 0 0) 0x7f0704160c50 con 0x7f0704029c30 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:39:06.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070402d810 con 0x7f0704029c30 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1704592605 0 0) 0x7f06f0004eb0 con 0x7f0704161170 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.882+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 >> v1:172.21.15.33:6790/0 conn(0x7f0704029c30 legacy=0x7f070402d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 >> v1:172.21.15.153:6789/0 conn(0x7f0704006a70 legacy=0x7f070402dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 1 monclient: found mon.a 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040cd0c0 con 0x7f0704161170 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f06f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 10 monclient: _renew_subs 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0704032370 con 0x7f0704161170 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:06.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 10 monclient: _renew_subs 2024-03-29T11:39:06.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:06.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07041615c0 con 0x7f0704161170 2024-03-29T11:39:06.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.883+0000 7f0711b9d640 1 librados: init done 2024-03-29T11:39:06.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.885+0000 7f0711b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.885+0000 7f0711b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.885+0000 7f0711b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:06.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.885+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07040c4a60 con 0x7f0704161170 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.887+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f070c04fdd0 con 0x7f0704161170 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.887+0000 7f06f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.887+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f070c067bf0 con 0x7f0704161170 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.887+0000 7f06f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.887+0000 7f06f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.887+0000 7f06f7fff640 10 monclient: dump: 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:06.890 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.887+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f070c067e10 con 0x7f0704161170 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.888+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f070c0ff410 con 0x7f0704161170 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.888+0000 7f06f7fff640 10 client.4386.objecter ms_dispatch 0x7f070c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.888+0000 7f06f7fff640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.888+0000 7f06f7fff640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.888+0000 7f06f7fff640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T11:39:06.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.888+0000 7f0709ffb640 4 set_mon_vals no callback set 2024-03-29T11:39:06.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.890+0000 7f070b7fe640 10 monclient: get_auth_request con 0x7f06f006df10 auth_method 0 2024-03-29T11:39:06.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.891+0000 7f06f7fff640 10 client.4386.objecter ms_handle_connect 0x7f06f006df10 2024-03-29T11:39:06.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.895+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f070c0cad70 con 0x7f0704161170 2024-03-29T11:39:06.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.895+0000 7f06f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:06.895+0000 7f06f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:07.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.237+0000 7f0711b9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:07.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.237+0000 7f0711b9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:07.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.237+0000 7f0711b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:07.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.237+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0704004cd0 con 0x7f0704161170 2024-03-29T11:39:07.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.238+0000 7f06f7fff640 1 -- 172.21.15.33:0/3116814394 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f070c0ccca0 con 0x7f0704161170 2024-03-29T11:39:07.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.238+0000 7f06f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:07.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.238+0000 7f06f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:07.240 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:07.240 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:07.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 10 librados: watch_flush enter 2024-03-29T11:39:07.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 10 librados: watch_flush exit 2024-03-29T11:39:07.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 10 client.4386.objecter successfully canceled tick 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 >> v1:172.21.15.153:6816/2737028074 conn(0x7f06f006df10 legacy=0x7f06f00703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 10 monclient: shutdown 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.241+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 >> v1:172.21.15.33:6789/0 conn(0x7f0704161170 legacy=0x7f0704031c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.242+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 shutdown_connections 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.242+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 >> 172.21.15.33:0/3116814394 conn(0x7f07040058f0 msgr2=0x7f07040c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.242+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 shutdown_connections 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.242+0000 7f0711b9d640 1 -- 172.21.15.33:0/3116814394 wait complete. 2024-03-29T11:39:07.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:07.242+0000 7f0711b9d640 1 librados: shutdown 2024-03-29T11:39:07.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:14.069 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.202+0000 7f95d3365640 10 monclient: get_monmap_and_config 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.202+0000 7f95d3365640 10 monclient: build_initial_monmap 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.202+0000 7f95d3365640 10 monclient: monmap: 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:14.204309+0000 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:14.204309+0000 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:14.204 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:14.205 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:14.205 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:14.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.203+0000 7f95d3365640 1 Processor -- start 2024-03-29T11:39:14.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.203+0000 7f95d3365640 1 -- start start 2024-03-29T11:39:14.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.203+0000 7f95d3365640 10 monclient: init 2024-03-29T11:39:14.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.204+0000 7f95d3365640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:14.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.204+0000 7f95d3365640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:14.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.204+0000 7f95d3365640 10 monclient(hunting): picked mon.noname-a con 0x7f95cc1457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:14.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.204+0000 7f95d3365640 10 monclient(hunting): picked mon.noname-b con 0x7f95cc13f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:14.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.204+0000 7f95d0900640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f95cc13f650 0x7f95cc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:45722/0 (socket says 172.21.15.33:45722) 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.204+0000 7f95d0900640 1 -- 172.21.15.33:0/1293369545 learned_addr learned my addr 172.21.15.33:0/1293369545 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95d3365640 10 monclient(hunting): picked mon.noname-c con 0x7f95cc0a9d70 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc0aaef0 con 0x7f95cc1457f0 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc145bd0 con 0x7f95cc0a9d70 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc145eb0 con 0x7f95cc13f650 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95d3365640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95d3365640 10 monclient(hunting): authenticate will time out at 1110.470171s 2024-03-29T11:39:14.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879777330 0 0) 0x7f95cc145eb0 con 0x7f95cc13f650 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95c3fff640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.205+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b8003520 con 0x7f95cc13f650 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705168593 0 0) 0x7f95cc145bd0 con 0x7f95cc0a9d70 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95cc145eb0 con 0x7f95cc0a9d70 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2084174334 0 0) 0x7f95b8003520 con 0x7f95cc13f650 2024-03-29T11:39:14.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 >> v1:172.21.15.153:6789/0 conn(0x7f95cc0a9d70 legacy=0x7f95cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.206+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 >> v1:172.21.15.33:6789/0 conn(0x7f95cc1457f0 legacy=0x7f95cc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:14.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:14.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 1 monclient: found mon.noname-b 2024-03-29T11:39:14.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:14.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc146340 con 0x7f95cc13f650 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95bc001690 con 0x7f95cc13f650 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: dump: 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:14.211 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95d3365640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95d3365640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95bc0017f0 con 0x7f95cc13f650 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95d3365640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95d3365640 4 set_mon_vals no callback set 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 1 -- 172.21.15.33:0/1293369545 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95bc001c00 con 0x7f95cc13f650 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.208+0000 7f95c3fff640 10 monclient: dump: 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:14.212 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.213 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.213 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.213 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:14.213 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:14.213 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:14.213 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:14.213 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.209+0000 7f95d3365640 10 monclient: shutdown 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.209+0000 7f95d3365640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.209+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 >> v1:172.21.15.33:6790/0 conn(0x7f95cc13f650 legacy=0x7f95cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.209+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 shutdown_connections 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.209+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 >> 172.21.15.33:0/1293369545 conn(0x7f95cc066a00 msgr2=0x7f95cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.209+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 shutdown_connections 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.209+0000 7f95d3365640 1 -- 172.21.15.33:0/1293369545 wait complete. 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.210+0000 7f95d3365640 10 monclient: build_initial_monmap 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.210+0000 7f95d3365640 10 monclient: monmap: 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:14.216 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.211+0000 7f95d3365640 1 librados: starting msgr at 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.211+0000 7f95d3365640 1 librados: starting objecter 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.211+0000 7f95d3365640 1 Processor -- start 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.211+0000 7f95d3365640 1 -- start start 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.211+0000 7f95d3365640 1 librados: setting wanted keys 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.211+0000 7f95d3365640 1 librados: calling monclient init 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.211+0000 7f95d3365640 10 monclient: init 2024-03-29T11:39:14.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 10 monclient(hunting): picked mon.noname-c con 0x7f95cc0a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 10 monclient(hunting): picked mon.noname-b con 0x7f95cc13f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 10 monclient(hunting): picked mon.noname-a con 0x7f95cc1457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc1387a0 con 0x7f95cc1457f0 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc0ac1a0 con 0x7f95cc13f650 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc0ad010 con 0x7f95cc0a9d70 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d3365640 10 monclient(hunting): authenticate will time out at 1110.477374s 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d1101640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f95cc0a9d70 0x7f95cc1d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:46966/0 (socket says 172.21.15.33:46966) 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.212+0000 7f95d1101640 1 -- 172.21.15.33:0/2648570181 learned_addr learned my addr 172.21.15.33:0/2648570181 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 10 client.?.objecter ms_handle_connect 0x7f95cc0a9d70 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173465963 0 0) 0x7f95cc0ad010 con 0x7f95cc0a9d70 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:14.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.213+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b0004eb0 con 0x7f95cc0a9d70 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2476222987 0 0) 0x7f95b0004eb0 con 0x7f95cc0a9d70 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95cc0ad010 con 0x7f95cc0a9d70 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95c80017d0 con 0x7f95cc0a9d70 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:14.219 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:14.220 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:14.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:14.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 972939721 0 0) 0x7f95cc0ad010 con 0x7f95cc0a9d70 2024-03-29T11:39:14.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 >> v1:172.21.15.153:6789/0 conn(0x7f95cc13f650 legacy=0x7f95cc137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.214+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 >> v1:172.21.15.33:6789/0 conn(0x7f95cc1457f0 legacy=0x7f95cc138090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:14.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:14.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 1 monclient: found mon.c 2024-03-29T11:39:14.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:14.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc1d7210 con 0x7f95cc0a9d70 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95d3365640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95d3365640 10 monclient: _renew_subs 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95d3365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.215+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc1d7e40 con 0x7f95cc0a9d70 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95c8001930 con 0x7f95cc0a9d70 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95c8004a00 con 0x7f95cc0a9d70 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95c1ffb640 10 monclient: dump: 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:14.223 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95d3365640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95c37fe640 4 set_mon_vals no callback set 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95d3365640 10 monclient: _renew_subs 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95d3365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc13ff80 con 0x7f95cc0a9d70 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.216+0000 7f95d3365640 1 librados: init done 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.217+0000 7f95d3365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.217+0000 7f95d3365640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:14.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.217+0000 7f95d3365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.217+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95cc1d7070 con 0x7f95cc0a9d70 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.218+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f95c801a9d0 con 0x7f95cc0a9d70 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.219+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f95c8084a40 con 0x7f95cc0a9d70 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.219+0000 7f95c1ffb640 10 client.4430.objecter ms_dispatch 0x7f95cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.219+0000 7f95c1ffb640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.219+0000 7f95c1ffb640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.219+0000 7f95c1ffb640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.220+0000 7f95d0900640 10 monclient: get_auth_request con 0x7f95b006dc40 auth_method 0 2024-03-29T11:39:14.225 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.220+0000 7f95c1ffb640 10 client.4430.objecter ms_handle_connect 0x7f95b006dc40 2024-03-29T11:39:14.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.226+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95cc1d7070 con 0x7f95cc0a9d70 2024-03-29T11:39:14.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.226+0000 7f95c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:14.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.226+0000 7f95c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:14.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.548+0000 7f95d3365640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:14.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.548+0000 7f95d3365640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:14.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.548+0000 7f95d3365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:14.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.548+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95cc1d7070 con 0x7f95cc0a9d70 2024-03-29T11:39:14.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.549+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2648570181 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f95cc1d7070 con 0x7f95cc0a9d70 2024-03-29T11:39:14.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.549+0000 7f95c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:14.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.549+0000 7f95c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:14.551 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:14.551 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.552+0000 7f95d3365640 10 librados: watch_flush enter 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.552+0000 7f95d3365640 10 librados: watch_flush exit 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.552+0000 7f95d3365640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.552+0000 7f95d3365640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.552+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 >> v1:172.21.15.153:6816/2737028074 conn(0x7f95b006dc40 legacy=0x7f95b0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 10 monclient: shutdown 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 >> v1:172.21.15.33:6790/0 conn(0x7f95cc0a9d70 legacy=0x7f95cc1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 shutdown_connections 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 >> 172.21.15.33:0/2648570181 conn(0x7f95cc066a00 msgr2=0x7f95cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 shutdown_connections 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 1 -- 172.21.15.33:0/2648570181 wait complete. 2024-03-29T11:39:14.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:14.553+0000 7f95d3365640 1 librados: shutdown 2024-03-29T11:39:14.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:21.375 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:21.514 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.512+0000 7f2a64932640 10 monclient: get_monmap_and_config 2024-03-29T11:39:21.514 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.512+0000 7f2a64932640 10 monclient: build_initial_monmap 2024-03-29T11:39:21.514 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.512+0000 7f2a64932640 10 monclient: monmap: 2024-03-29T11:39:21.514 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:21.514516+0000 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:21.514516+0000 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.513+0000 7f2a64932640 1 Processor -- start 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.513+0000 7f2a64932640 1 -- start start 2024-03-29T11:39:21.515 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.513+0000 7f2a64932640 10 monclient: init 2024-03-29T11:39:21.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:21.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:21.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 10 monclient(hunting): picked mon.noname-b con 0x7f2a600b4330 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:21.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 10 monclient(hunting): picked mon.noname-c con 0x7f2a601470b0 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:21.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 10 monclient(hunting): picked mon.noname-a con 0x7f2a601439b0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:21.517 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6006f2a0 con 0x7f2a601439b0 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a60146b90 con 0x7f2a601470b0 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a600b2ca0 con 0x7f2a600b4330 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a64932640 10 monclient(hunting): authenticate will time out at 1117.779787s 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5e59c640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f2a600b4330 0x7f2a600b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:50836/0 (socket says 172.21.15.33:50836) 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5e59c640 1 -- 172.21.15.33:0/2750991395 learned_addr learned my addr 172.21.15.33:0/2750991395 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407318083 0 0) 0x7f2a600b2ca0 con 0x7f2a600b4330 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5d59a640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.515+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a48003520 con 0x7f2a600b4330 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.516+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1798651773 0 0) 0x7f2a48003520 con 0x7f2a600b4330 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.516+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 >> v1:172.21.15.153:6789/0 conn(0x7f2a601470b0 legacy=0x7f2a600b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:21.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.516+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 >> v1:172.21.15.33:6789/0 conn(0x7f2a601439b0 legacy=0x7f2a60151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 1 monclient: found mon.noname-b 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6006e0c0 con 0x7f2a600b4330 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: _finish_auth 0 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:21.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a64932640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a64932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2a54001690 con 0x7f2a600b4330 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: dump: 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a64932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:21.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a540019a0 con 0x7f2a600b4330 2024-03-29T11:39:21.521 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a5d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a64932640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.517+0000 7f2a64932640 4 set_mon_vals no callback set 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.518+0000 7f2a5d59a640 1 -- 172.21.15.33:0/2750991395 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2a54001d70 con 0x7f2a600b4330 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.518+0000 7f2a5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.518+0000 7f2a5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.518+0000 7f2a5d59a640 10 monclient: dump: 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:21.523 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.518+0000 7f2a64932640 10 monclient: shutdown 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.518+0000 7f2a64932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.518+0000 7f2a64932640 1 -- 172.21.15.33:0/2750991395 >> v1:172.21.15.33:6790/0 conn(0x7f2a600b4330 legacy=0x7f2a600b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.519+0000 7f2a64932640 1 -- 172.21.15.33:0/2750991395 shutdown_connections 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.519+0000 7f2a64932640 1 -- 172.21.15.33:0/2750991395 >> 172.21.15.33:0/2750991395 conn(0x7f2a60066a00 msgr2=0x7f2a60066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.519+0000 7f2a64932640 1 -- 172.21.15.33:0/2750991395 shutdown_connections 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.519+0000 7f2a64932640 1 -- 172.21.15.33:0/2750991395 wait complete. 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 10 monclient: build_initial_monmap 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 10 monclient: monmap: 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:21.524 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 1 librados: starting msgr at 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 1 librados: starting objecter 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 1 Processor -- start 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 1 -- start start 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 1 librados: setting wanted keys 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 1 librados: calling monclient init 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.520+0000 7f2a64932640 10 monclient: init 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 10 monclient(hunting): picked mon.noname-b con 0x7f2a600b4330 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:21.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 10 monclient(hunting): picked mon.noname-a con 0x7f2a601439b0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:21.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 10 monclient(hunting): picked mon.noname-c con 0x7f2a601470b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:21.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6011f910 con 0x7f2a601439b0 2024-03-29T11:39:21.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a601465e0 con 0x7f2a600b4330 2024-03-29T11:39:21.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a601204b0 con 0x7f2a601470b0 2024-03-29T11:39:21.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:21.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.521+0000 7f2a64932640 10 monclient(hunting): authenticate will time out at 1117.786481s 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a5ed9d640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f2a601470b0 0x7f2a6011fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:58182/0 (socket says 172.21.15.33:58182) 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a5ed9d640 1 -- 172.21.15.33:0/2331097088 learned_addr learned my addr 172.21.15.33:0/2331097088 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 10 client.?.objecter ms_handle_connect 0x7f2a601470b0 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869649554 0 0) 0x7f2a601204b0 con 0x7f2a601470b0 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.522+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a40004eb0 con 0x7f2a601470b0 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 368338082 0 0) 0x7f2a40004eb0 con 0x7f2a601470b0 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a601204b0 con 0x7f2a601470b0 2024-03-29T11:39:21.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2a58001ad0 con 0x7f2a601470b0 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 10 monclient(hunting): dump: 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1888949983 0 0) 0x7f2a601204b0 con 0x7f2a601470b0 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 >> v1:172.21.15.153:6789/0 conn(0x7f2a600b4330 legacy=0x7f2a601dee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.531 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.523+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 >> v1:172.21.15.33:6789/0 conn(0x7f2a601439b0 legacy=0x7f2a601df540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.524+0000 7f2a4effd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.524+0000 7f2a4effd640 1 monclient: found mon.c 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.524+0000 7f2a4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.524+0000 7f2a4effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.524+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a60120640 con 0x7f2a601470b0 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.524+0000 7f2a4effd640 10 monclient: _finish_auth 0 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.525+0000 7f2a4effd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.525+0000 7f2a4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.525+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a58004be0 con 0x7f2a601470b0 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.525+0000 7f2a4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.525+0000 7f2a5cd99640 4 set_mon_vals no callback set 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.525+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2a58004ef0 con 0x7f2a601470b0 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a4effd640 10 monclient: dump: 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:21.532 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 10 monclient: _renew_subs 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a60120890 con 0x7f2a601470b0 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 10 monclient: _renew_subs 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a601e7360 con 0x7f2a601470b0 2024-03-29T11:39:21.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.526+0000 7f2a64932640 1 librados: init done 2024-03-29T11:39:21.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.528+0000 7f2a64932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:21.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.528+0000 7f2a64932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:21.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.528+0000 7f2a64932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:21.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.528+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a60075f10 con 0x7f2a601470b0 2024-03-29T11:39:21.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.528+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f2a5801aec0 con 0x7f2a601470b0 2024-03-29T11:39:21.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.529+0000 7f2a5e59c640 10 monclient: get_auth_request con 0x7f2a4006dbc0 auth_method 0 2024-03-29T11:39:21.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.530+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f2a58084de0 con 0x7f2a601470b0 2024-03-29T11:39:21.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.530+0000 7f2a4effd640 10 client.4436.objecter ms_dispatch 0x7f2a60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:21.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.530+0000 7f2a4effd640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:21.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.530+0000 7f2a4effd640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:21.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.530+0000 7f2a4effd640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:39:21.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.530+0000 7f2a4effd640 10 client.4436.objecter ms_handle_connect 0x7f2a4006dbc0 2024-03-29T11:39:21.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.536+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a58050720 con 0x7f2a601470b0 2024-03-29T11:39:21.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.536+0000 7f2a4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:21.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.536+0000 7f2a4effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:21.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.857+0000 7f2a64932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:21.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.857+0000 7f2a64932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:21.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.857+0000 7f2a64932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:21.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.857+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a60071690 con 0x7f2a601470b0 2024-03-29T11:39:21.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.858+0000 7f2a4effd640 1 -- 172.21.15.33:0/2331097088 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f2a58084490 con 0x7f2a601470b0 2024-03-29T11:39:21.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.858+0000 7f2a4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:21.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.858+0000 7f2a4effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:21.860 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:21.860 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:21.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 10 librados: watch_flush enter 2024-03-29T11:39:21.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 10 librados: watch_flush exit 2024-03-29T11:39:21.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:39:21.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 >> v1:172.21.15.153:6816/2737028074 conn(0x7f2a4006dbc0 legacy=0x7f2a40070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 10 monclient: shutdown 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 >> v1:172.21.15.33:6790/0 conn(0x7f2a601470b0 legacy=0x7f2a6011fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 shutdown_connections 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 >> 172.21.15.33:0/2331097088 conn(0x7f2a60066a00 msgr2=0x7f2a600b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.861+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 shutdown_connections 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.862+0000 7f2a64932640 1 -- 172.21.15.33:0/2331097088 wait complete. 2024-03-29T11:39:21.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:21.862+0000 7f2a64932640 1 librados: shutdown 2024-03-29T11:39:21.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:28.684 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:28.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.833+0000 7f944b67b640 10 monclient: get_monmap_and_config 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.833+0000 7f944b67b640 10 monclient: build_initial_monmap 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.834+0000 7f944b67b640 10 monclient: monmap: 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:28.835599+0000 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:28.835599+0000 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:28.836 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:28.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.835+0000 7f944b67b640 1 Processor -- start 2024-03-29T11:39:28.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.835+0000 7f944b67b640 1 -- start start 2024-03-29T11:39:28.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.835+0000 7f944b67b640 10 monclient: init 2024-03-29T11:39:28.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:28.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 10 monclient(hunting): picked mon.noname-b con 0x7f94440b4330 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 10 monclient(hunting): picked mon.noname-c con 0x7f94441470b0 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 10 monclient(hunting): picked mon.noname-a con 0x7f94441439b0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944406f2a0 con 0x7f94441439b0 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9444146b90 con 0x7f94441470b0 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440b2ca0 con 0x7f94440b4330 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:28.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.836+0000 7f944b67b640 10 monclient(hunting): authenticate will time out at 1125.101056s 2024-03-29T11:39:28.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.839+0000 7f9448c16640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f94441470b0 0x7f94440b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:43897/0 (socket says 172.21.15.33:43897) 2024-03-29T11:39:28.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.839+0000 7f9448c16640 1 -- 172.21.15.33:0/1804610531 learned_addr learned my addr 172.21.15.33:0/1804610531 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:28.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121143134 0 0) 0x7f944406f2a0 con 0x7f94441439b0 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9430003520 con 0x7f94441439b0 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466350078 0 0) 0x7f9444146b90 con 0x7f94441470b0 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:39:28.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f944406f2a0 con 0x7f94441470b0 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 315123912 0 0) 0x7f9430003520 con 0x7f94441439b0 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 >> v1:172.21.15.33:6790/0 conn(0x7f94440b4330 legacy=0x7f94440b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 >> v1:172.21.15.153:6789/0 conn(0x7f94441470b0 legacy=0x7f94440b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 monclient: found mon.noname-a 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944406e0c0 con 0x7f94441439b0 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: _finish_auth 0 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:28.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f943c001a10 con 0x7f94441439b0 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: dump: 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:28.846 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f943c001b70 con 0x7f94441439b0 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 1 -- 172.21.15.33:0/1804610531 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f943c001f80 con 0x7f94441439b0 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f943bfff640 10 monclient: dump: 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:28.847 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:28.848 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:28.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.842+0000 7f944b67b640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:39:28.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.843+0000 7f944b67b640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:28.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.843+0000 7f944b67b640 4 set_mon_vals no callback set 2024-03-29T11:39:28.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.843+0000 7f944b67b640 10 monclient: shutdown 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.843+0000 7f944b67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.843+0000 7f944b67b640 1 -- 172.21.15.33:0/1804610531 >> v1:172.21.15.33:6789/0 conn(0x7f94441439b0 legacy=0x7f9444151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.844+0000 7f944b67b640 1 -- 172.21.15.33:0/1804610531 shutdown_connections 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.844+0000 7f944b67b640 1 -- 172.21.15.33:0/1804610531 >> 172.21.15.33:0/1804610531 conn(0x7f9444066a00 msgr2=0x7f9444066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.845+0000 7f944b67b640 1 -- 172.21.15.33:0/1804610531 shutdown_connections 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.845+0000 7f944b67b640 1 -- 172.21.15.33:0/1804610531 wait complete. 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 10 monclient: build_initial_monmap 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 10 monclient: monmap: 2024-03-29T11:39:28.849 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 1 librados: starting msgr at 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 1 librados: starting objecter 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 1 Processor -- start 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 1 -- start start 2024-03-29T11:39:28.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 1 librados: setting wanted keys 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 1 librados: calling monclient init 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.846+0000 7f944b67b640 10 monclient: init 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 10 monclient(hunting): picked mon.noname-c con 0x7f94441439b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 10 monclient(hunting): picked mon.noname-a con 0x7f94441470b0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 10 monclient(hunting): picked mon.noname-b con 0x7f94440aacd0 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440ad050 con 0x7f94441470b0 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440abc90 con 0x7f94440aacd0 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440ab6e0 con 0x7f94441439b0 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:28.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.847+0000 7f944b67b640 10 monclient(hunting): authenticate will time out at 1125.112386s 2024-03-29T11:39:28.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.852+0000 7f9448c16640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f94441470b0 0x7f94441e72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:59495/0 (socket says 172.21.15.33:59495) 2024-03-29T11:39:28.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.852+0000 7f9448c16640 1 -- 172.21.15.33:0/3008939111 learned_addr learned my addr 172.21.15.33:0/3008939111 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:28.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 client.?.objecter ms_handle_connect 0x7f94441470b0 2024-03-29T11:39:28.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:28.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 client.?.objecter ms_handle_connect 0x7f94441439b0 2024-03-29T11:39:28.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:28.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198126701 0 0) 0x7f94440ad050 con 0x7f94441470b0 2024-03-29T11:39:28.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9428004eb0 con 0x7f94441470b0 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741546845 0 0) 0x7f94440ab6e0 con 0x7f94441439b0 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9428006230 con 0x7f94441439b0 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 client.?.objecter ms_handle_connect 0x7f94440aacd0 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.854+0000 7f9439ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516564362 0 0) 0x7f94440abc90 con 0x7f94440aacd0 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94440ab6e0 con 0x7f94440aacd0 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895351222 0 0) 0x7f9428006230 con 0x7f94441439b0 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9428006880 con 0x7f94441439b0 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9434001760 con 0x7f94441439b0 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.861 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.855+0000 7f9439ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1614619477 0 0) 0x7f9428004eb0 con 0x7f94441470b0 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9428006230 con 0x7f94441470b0 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9440001970 con 0x7f94441470b0 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:28.862 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3010034174 0 0) 0x7f9428006880 con 0x7f94441439b0 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 >> v1:172.21.15.153:6789/0 conn(0x7f94440aacd0 legacy=0x7f94441ef680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 >> v1:172.21.15.33:6789/0 conn(0x7f94441470b0 legacy=0x7f94441e72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 monclient: found mon.c 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94440ab870 con 0x7f94441439b0 2024-03-29T11:39:28.863 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f9439ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f944b67b640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f944b67b640 10 monclient: _renew_subs 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f944b67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.856+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94441d0c80 con 0x7f94441439b0 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.857+0000 7f944b67b640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.857+0000 7f944b67b640 10 monclient: _renew_subs 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.857+0000 7f944b67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.857+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94440b1430 con 0x7f94441439b0 2024-03-29T11:39:28.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.857+0000 7f944b67b640 1 librados: init done 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.859+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9434001c70 con 0x7f94441439b0 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.859+0000 7f9439ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.859+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9434003790 con 0x7f94441439b0 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.859+0000 7f9439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.859+0000 7f9439ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.859+0000 7f9439ffb640 10 monclient: dump: 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:28.867 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.860+0000 7f943affd640 4 set_mon_vals no callback set 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.860+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f9434019760 con 0x7f94441439b0 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f9434083660 con 0x7f94441439b0 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f9439ffb640 10 client.4439.objecter ms_dispatch 0x7f9444000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f9439ffb640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f9439ffb640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f9439ffb640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:39:28.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f944b67b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:28.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f944b67b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:28.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f944b67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:28.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9424005080 con 0x7f94441439b0 2024-03-29T11:39:28.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.861+0000 7f9448c16640 10 monclient: get_auth_request con 0x7f942806e570 auth_method 0 2024-03-29T11:39:28.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.863+0000 7f9439ffb640 10 client.4439.objecter ms_handle_connect 0x7f942806e570 2024-03-29T11:39:28.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.869+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f943404ed70 con 0x7f94441439b0 2024-03-29T11:39:28.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.869+0000 7f9439ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:28.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:28.869+0000 7f9439ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:29.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.203+0000 7f944b67b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:29.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.203+0000 7f944b67b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:29.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.203+0000 7f944b67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:29.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.203+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9424005330 con 0x7f94441439b0 2024-03-29T11:39:29.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.204+0000 7f9439ffb640 1 -- 172.21.15.33:0/3008939111 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f9434050e70 con 0x7f94441439b0 2024-03-29T11:39:29.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.204+0000 7f9439ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:29.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.204+0000 7f9439ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:29.209 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:29.209 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:29.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.207+0000 7f944b67b640 10 librados: watch_flush enter 2024-03-29T11:39:29.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 10 librados: watch_flush exit 2024-03-29T11:39:29.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:39:29.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 >> v1:172.21.15.153:6816/2737028074 conn(0x7f942806e570 legacy=0x7f9428070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 10 monclient: shutdown 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 >> v1:172.21.15.33:6790/0 conn(0x7f94441439b0 legacy=0x7f94440b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 shutdown_connections 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.208+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 >> 172.21.15.33:0/3008939111 conn(0x7f9444066a00 msgr2=0x7f94440705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.209+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 shutdown_connections 2024-03-29T11:39:29.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.209+0000 7f944b67b640 1 -- 172.21.15.33:0/3008939111 wait complete. 2024-03-29T11:39:29.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:29.209+0000 7f944b67b640 1 librados: shutdown 2024-03-29T11:39:29.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:36.035 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:36.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.181+0000 7f986c10e640 10 monclient: get_monmap_and_config 2024-03-29T11:39:36.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.181+0000 7f986c10e640 10 monclient: build_initial_monmap 2024-03-29T11:39:36.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.181+0000 7f986c10e640 10 monclient: monmap: 2024-03-29T11:39:36.182 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:36.182 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:36.182158+0000 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:36.182158+0000 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:36.183 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.182+0000 7f986c10e640 1 Processor -- start 2024-03-29T11:39:36.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.182+0000 7f986c10e640 1 -- start start 2024-03-29T11:39:36.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.182+0000 7f986c10e640 10 monclient: init 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 10 monclient(hunting): picked mon.noname-c con 0x7f985c038c60 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 10 monclient(hunting): picked mon.noname-b con 0x7f985c0324b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 10 monclient(hunting): picked mon.noname-a con 0x7f985c0d11e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c0caf80 con 0x7f985c0d11e0 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c039850 con 0x7f985c038c60 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c0c78d0 con 0x7f985c0324b0 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:36.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.183+0000 7f986c10e640 10 monclient(hunting): authenticate will time out at 1132.447585s 2024-03-29T11:39:36.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.188+0000 7f9869eaa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f985c038c60 0x7f985c0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:36215/0 (socket says 172.21.15.33:36215) 2024-03-29T11:39:36.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.188+0000 7f9869eaa640 1 -- 172.21.15.33:0/474872136 learned_addr learned my addr 172.21.15.33:0/474872136 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:36.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.190+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401523979 0 0) 0x7f985c0c78d0 con 0x7f985c0324b0 2024-03-29T11:39:36.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.190+0000 7f9868ea8640 10 monclient(hunting): _negotiate 2024-03-29T11:39:36.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.190+0000 7f9868ea8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:36.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.190+0000 7f9868ea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:36.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.190+0000 7f9868ea8640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:39:36.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.190+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9850003520 con 0x7f985c0324b0 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195432572 0 0) 0x7f985c039850 con 0x7f985c038c60 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient(hunting): _negotiate 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f985c0c78d0 con 0x7f985c038c60 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4229019664 0 0) 0x7f9850003520 con 0x7f985c0324b0 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 >> v1:172.21.15.153:6789/0 conn(0x7f985c038c60 legacy=0x7f985c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 >> v1:172.21.15.33:6789/0 conn(0x7f985c0d11e0 legacy=0x7f985c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient: _finish_hunting 0 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 monclient: found mon.noname-b 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c0c7a60 con 0x7f985c0324b0 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient: _finish_auth 0 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient: _check_auth_tickets 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f986c10e640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f986c10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9854001690 con 0x7f985c0324b0 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.192+0000 7f9868ea8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.193+0000 7f9868ea8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:36.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.193+0000 7f9868ea8640 10 monclient: dump: 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.193+0000 7f986c10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.193+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98540018f0 con 0x7f985c0324b0 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.193+0000 7f9868ea8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:36.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.193+0000 7f986c10e640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.193+0000 7f986c10e640 4 set_mon_vals no callback set 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f986c10e640 10 monclient: shutdown 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f986c10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f986c10e640 1 -- 172.21.15.33:0/474872136 >> v1:172.21.15.33:6790/0 conn(0x7f985c0324b0 legacy=0x7f985c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f9868ea8640 1 -- 172.21.15.33:0/474872136 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9854001690 con 0x7f985c0324b0 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f9868ea8640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f986c10e640 1 -- 172.21.15.33:0/474872136 shutdown_connections 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f986c10e640 1 -- 172.21.15.33:0/474872136 >> 172.21.15.33:0/474872136 conn(0x7f985c005c20 msgr2=0x7f985c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f986c10e640 1 -- 172.21.15.33:0/474872136 shutdown_connections 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.194+0000 7f986c10e640 1 -- 172.21.15.33:0/474872136 wait complete. 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.195+0000 7f986c10e640 10 monclient: build_initial_monmap 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.195+0000 7f986c10e640 10 monclient: monmap: 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:36.202 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.196+0000 7f986c10e640 1 librados: starting msgr at 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.196+0000 7f986c10e640 1 librados: starting objecter 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.196+0000 7f986c10e640 1 Processor -- start 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.196+0000 7f986c10e640 1 -- start start 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.196+0000 7f986c10e640 1 librados: setting wanted keys 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.196+0000 7f986c10e640 1 librados: calling monclient init 2024-03-29T11:39:36.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.196+0000 7f986c10e640 10 monclient: init 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 10 monclient(hunting): picked mon.noname-b con 0x7f985c0324b0 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 10 monclient(hunting): picked mon.noname-c con 0x7f985c038c60 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 10 monclient(hunting): picked mon.noname-a con 0x7f985c0d11e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c02f360 con 0x7f985c0d11e0 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c0373a0 con 0x7f985c0324b0 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c158500 con 0x7f985c038c60 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:36.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.197+0000 7f986c10e640 10 monclient(hunting): authenticate will time out at 1132.461107s 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.201+0000 7f98696a9640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f985c038c60 0x7f985c1589e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:49197/0 (socket says 172.21.15.33:49197) 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.201+0000 7f98696a9640 1 -- 172.21.15.33:0/1123726091 learned_addr learned my addr 172.21.15.33:0/1123726091 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.203+0000 7f985a7fc640 10 client.?.objecter ms_handle_connect 0x7f985c0324b0 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.203+0000 7f985a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.203+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768664360 0 0) 0x7f985c0373a0 con 0x7f985c0324b0 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.204+0000 7f985a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.204+0000 7f985a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.204+0000 7f985a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.204+0000 7f985a7fc640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.204+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9848004eb0 con 0x7f985c0324b0 2024-03-29T11:39:36.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.204+0000 7f985a7fc640 10 client.?.objecter ms_handle_connect 0x7f985c0d11e0 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.204+0000 7f985a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2062147420 0 0) 0x7f9848004eb0 con 0x7f985c0324b0 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9848005300 con 0x7f985c0324b0 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f986404fdd0 con 0x7f985c0324b0 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): dump: 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:36.207 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:36.208 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:36.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:36.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 client.?.objecter ms_handle_connect 0x7f985c038c60 2024-03-29T11:39:36.208 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394368034 0 0) 0x7f985c158500 con 0x7f985c038c60 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9848005850 con 0x7f985c038c60 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3832254892 0 0) 0x7f9848005300 con 0x7f985c0324b0 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 >> v1:172.21.15.33:6790/0 conn(0x7f985c038c60 legacy=0x7f985c1589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.210 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 >> v1:172.21.15.33:6789/0 conn(0x7f985c0d11e0 legacy=0x7f985c02d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 monclient: found mon.b 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c02f5d0 con 0x7f985c0324b0 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f985a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f986c10e640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f986c10e640 10 monclient: _renew_subs 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f986c10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.205+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c00d560 con 0x7f985c0324b0 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.206+0000 7f986c10e640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.206+0000 7f986c10e640 10 monclient: _renew_subs 2024-03-29T11:39:36.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.206+0000 7f986c10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.206+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c02d350 con 0x7f985c0324b0 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.206+0000 7f986c10e640 1 librados: init done 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f986404f400 con 0x7f985c0324b0 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f985a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f986404fdd0 con 0x7f985c0324b0 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f985a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f985a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f985a7fc640 10 monclient: dump: 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:36.212 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:36.213 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:36.213 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:36.213 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:36.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f986c10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:36.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f986c10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:36.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.207+0000 7f986c10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:36.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.208+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c007b90 con 0x7f985c0324b0 2024-03-29T11:39:36.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.209+0000 7f985bfff640 4 set_mon_vals no callback set 2024-03-29T11:39:36.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.210+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f9864067bf0 con 0x7f985c0324b0 2024-03-29T11:39:36.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.211+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f98640ff220 con 0x7f985c0324b0 2024-03-29T11:39:36.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.211+0000 7f985a7fc640 10 client.4402.objecter ms_dispatch 0x7f9864000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:36.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.212+0000 7f985a7fc640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:36.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.212+0000 7f985a7fc640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:36.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.212+0000 7f985a7fc640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T11:39:36.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.212+0000 7f98696a9640 10 monclient: get_auth_request con 0x7f984806e1c0 auth_method 0 2024-03-29T11:39:36.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.213+0000 7f985a7fc640 10 client.4402.objecter ms_handle_connect 0x7f984806e1c0 2024-03-29T11:39:36.221 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.220+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98640cab00 con 0x7f985c0324b0 2024-03-29T11:39:36.221 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.220+0000 7f985a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:36.221 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.220+0000 7f985a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:36.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.549+0000 7f986c10e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:36.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.549+0000 7f986c10e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:36.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.549+0000 7f986c10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:36.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.549+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f985c007b90 con 0x7f985c0324b0 2024-03-29T11:39:36.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.550+0000 7f985a7fc640 1 -- 172.21.15.33:0/1123726091 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f98640cca30 con 0x7f985c0324b0 2024-03-29T11:39:36.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.550+0000 7f985a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:36.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.550+0000 7f985a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:36.555 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:36.555 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:36.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.554+0000 7f986c10e640 10 librados: watch_flush enter 2024-03-29T11:39:36.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.554+0000 7f986c10e640 10 librados: watch_flush exit 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.554+0000 7f986c10e640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.554+0000 7f986c10e640 10 client.4402.objecter successfully canceled tick 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.554+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 >> v1:172.21.15.153:6816/2737028074 conn(0x7f984806e1c0 legacy=0x7f9848070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 10 monclient: shutdown 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 >> v1:172.21.15.153:6789/0 conn(0x7f985c0324b0 legacy=0x7f985c02fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 shutdown_connections 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 >> 172.21.15.33:0/1123726091 conn(0x7f985c005c20 msgr2=0x7f985c03ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:36.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 shutdown_connections 2024-03-29T11:39:36.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 1 -- 172.21.15.33:0/1123726091 wait complete. 2024-03-29T11:39:36.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:36.555+0000 7f986c10e640 1 librados: shutdown 2024-03-29T11:39:36.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:43.379 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.524+0000 7f0af5b6d640 10 monclient: get_monmap_and_config 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.524+0000 7f0af5b6d640 10 monclient: build_initial_monmap 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.524+0000 7f0af5b6d640 10 monclient: monmap: 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:43.524739+0000 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:43.524739+0000 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:43.525 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:43.526 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:43.526 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:43.526 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:43.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.525+0000 7f0af5b6d640 1 Processor -- start 2024-03-29T11:39:43.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.525+0000 7f0af5b6d640 1 -- start start 2024-03-29T11:39:43.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.525+0000 7f0af5b6d640 10 monclient: init 2024-03-29T11:39:43.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.526+0000 7f0af5b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:43.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.527+0000 7f0af5b6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:43.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.527+0000 7f0af5b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae80389e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:43.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.528+0000 7f0af5b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae8038dc0 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:43.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.528+0000 7f0af5b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae8030660 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:43.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.528+0000 7f0af5b6d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae80c54c0 con 0x7f0ae80389e0 2024-03-29T11:39:43.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.528+0000 7f0af5b6d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae8036430 con 0x7f0ae8030660 2024-03-29T11:39:43.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.528+0000 7f0af5b6d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae8039840 con 0x7f0ae8038dc0 2024-03-29T11:39:43.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.528+0000 7f0af5b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:43.529 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.528+0000 7f0af5b6d640 10 monclient(hunting): authenticate will time out at 1139.791790s 2024-03-29T11:39:43.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.529+0000 7f0aeffff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f0ae8038dc0 0x7f0ae80cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:38349/0 (socket says 172.21.15.33:38349) 2024-03-29T11:39:43.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.530+0000 7f0aeffff640 1 -- 172.21.15.33:0/3515259546 learned_addr learned my addr 172.21.15.33:0/3515259546 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:43.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.531+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192661550 0 0) 0x7f0ae8039840 con 0x7f0ae8038dc0 2024-03-29T11:39:43.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.531+0000 7f0aef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:43.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.531+0000 7f0aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:43.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.531+0000 7f0aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:43.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.531+0000 7f0aef7fe640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:39:43.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.531+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad8003520 con 0x7f0ae8038dc0 2024-03-29T11:39:43.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2487307837 0 0) 0x7f0ad8003520 con 0x7f0ae8038dc0 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 >> v1:172.21.15.153:6789/0 conn(0x7f0ae8030660 legacy=0x7f0ae80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 >> v1:172.21.15.33:6789/0 conn(0x7f0ae80389e0 legacy=0x7f0ae8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae8035690 con 0x7f0ae8038dc0 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0ae4001690 con 0x7f0ae8038dc0 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.533+0000 7f0aef7fe640 10 monclient: dump: 2024-03-29T11:39:43.537 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0af5b6d640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0af5b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ae4001920 con 0x7f0ae8038dc0 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0aef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0aef7fe640 1 -- 172.21.15.33:0/3515259546 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0ae4001cb0 con 0x7f0ae8038dc0 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0af5b6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:43.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0af5b6d640 4 set_mon_vals no callback set 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.534+0000 7f0aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.535+0000 7f0aef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.535+0000 7f0aef7fe640 10 monclient: dump: 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.536+0000 7f0af5b6d640 10 monclient: shutdown 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.536+0000 7f0af5b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:43.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.536+0000 7f0af5b6d640 1 -- 172.21.15.33:0/3515259546 >> v1:172.21.15.33:6790/0 conn(0x7f0ae8038dc0 legacy=0x7f0ae80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.538+0000 7f0af5b6d640 1 -- 172.21.15.33:0/3515259546 shutdown_connections 2024-03-29T11:39:43.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.538+0000 7f0af5b6d640 1 -- 172.21.15.33:0/3515259546 >> 172.21.15.33:0/3515259546 conn(0x7f0ae8005c00 msgr2=0x7f0ae8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:43.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.539+0000 7f0af5b6d640 1 -- 172.21.15.33:0/3515259546 shutdown_connections 2024-03-29T11:39:43.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.539+0000 7f0af5b6d640 1 -- 172.21.15.33:0/3515259546 wait complete. 2024-03-29T11:39:43.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.540+0000 7f0af5b6d640 10 monclient: build_initial_monmap 2024-03-29T11:39:43.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.541+0000 7f0af5b6d640 10 monclient: monmap: 2024-03-29T11:39:43.541 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:43.541 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:43.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.541+0000 7f0af5b6d640 1 librados: starting msgr at 2024-03-29T11:39:43.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.542+0000 7f0af5b6d640 1 librados: starting objecter 2024-03-29T11:39:43.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.542+0000 7f0af5b6d640 1 Processor -- start 2024-03-29T11:39:43.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.543+0000 7f0af5b6d640 1 -- start start 2024-03-29T11:39:43.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.543+0000 7f0af5b6d640 1 librados: setting wanted keys 2024-03-29T11:39:43.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.543+0000 7f0af5b6d640 1 librados: calling monclient init 2024-03-29T11:39:43.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.543+0000 7f0af5b6d640 10 monclient: init 2024-03-29T11:39:43.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.544+0000 7f0af5b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:43.545 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.544+0000 7f0af5b6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:43.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.544+0000 7f0af5b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae8030660 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:43.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af5b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae80389e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:43.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af5b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae80c1790 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:43.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af5b6d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae802f6e0 con 0x7f0ae80389e0 2024-03-29T11:39:43.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af5b6d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae802dde0 con 0x7f0ae80c1790 2024-03-29T11:39:43.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af5b6d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae80c1be0 con 0x7f0ae8030660 2024-03-29T11:39:43.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af5b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:43.549 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af5b6d640 10 monclient(hunting): authenticate will time out at 1139.811503s 2024-03-29T11:39:43.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.547+0000 7f0af4b6b640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f0ae8030660 0x7f0ae802e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:38453/0 (socket says 172.21.15.33:38453) 2024-03-29T11:39:43.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.553+0000 7f0aeffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f0ae80389e0 0x7f0ae80c1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49053/0 (socket says 172.21.15.33:49053) 2024-03-29T11:39:43.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.553+0000 7f0aeffff640 1 -- 172.21.15.33:0/1165735055 learned_addr learned my addr 172.21.15.33:0/1165735055 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:43.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.553+0000 7f0aed7fa640 10 client.?.objecter ms_handle_connect 0x7f0ae80389e0 2024-03-29T11:39:43.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.553+0000 7f0aed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 10 client.?.objecter ms_handle_connect 0x7f0ae80c1790 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114387331 0 0) 0x7f0ae802dde0 con 0x7f0ae80c1790 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:39:43.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.554+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0adc004eb0 con 0x7f0ae80c1790 2024-03-29T11:39:43.556 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.555+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3973270258 0 0) 0x7f0adc004eb0 con 0x7f0ae80c1790 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ae802dde0 con 0x7f0ae80c1790 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0af004fdd0 con 0x7f0ae80c1790 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient(hunting): dump: 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:43.557 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3479102836 0 0) 0x7f0ae802dde0 con 0x7f0ae80c1790 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 >> v1:172.21.15.33:6790/0 conn(0x7f0ae8030660 legacy=0x7f0ae802e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 >> v1:172.21.15.33:6789/0 conn(0x7f0ae80389e0 legacy=0x7f0ae80c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 1 monclient: found mon.b 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae802f910 con 0x7f0ae80c1790 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:43.559 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0aed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0af5b6d640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0af5b6d640 10 monclient: _renew_subs 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.556+0000 7f0af5b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.557+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae80c1da0 con 0x7f0ae80c1790 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.557+0000 7f0af5b6d640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.557+0000 7f0af5b6d640 10 monclient: _renew_subs 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.557+0000 7f0af5b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.557+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae815cf50 con 0x7f0ae80c1790 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.557+0000 7f0af5b6d640 1 librados: init done 2024-03-29T11:39:43.560 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.558+0000 7f0af5b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:43.561 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.558+0000 7f0af5b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:43.561 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.558+0000 7f0af5b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:43.561 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.558+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae8007b90 con 0x7f0ae80c1790 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.560+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0af004f400 con 0x7f0ae80c1790 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.561+0000 7f0aed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f0af0075f10 con 0x7f0ae80c1790 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 10 monclient: dump: 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.563 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f0af0067bf0 con 0x7f0ae80c1790 2024-03-29T11:39:43.564 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aeeffd640 4 set_mon_vals no callback set 2024-03-29T11:39:43.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f0af00fe3b0 con 0x7f0ae80c1790 2024-03-29T11:39:43.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 10 client.4408.objecter ms_dispatch 0x7f0af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:43.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:43.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:43.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.562+0000 7f0aed7fa640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-29T11:39:43.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.563+0000 7f0af4b6b640 10 monclient: get_auth_request con 0x7f0adc06d9b0 auth_method 0 2024-03-29T11:39:43.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.563+0000 7f0aed7fa640 10 client.4408.objecter ms_handle_connect 0x7f0adc06d9b0 2024-03-29T11:39:43.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.568+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0af00c9d70 con 0x7f0ae80c1790 2024-03-29T11:39:43.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.568+0000 7f0aed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:43.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.568+0000 7f0aed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:43.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.913+0000 7f0af5b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:43.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.913+0000 7f0af5b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:43.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.913+0000 7f0af5b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:39:43.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.913+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ae80c7af0 con 0x7f0ae80c1790 2024-03-29T11:39:43.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.915+0000 7f0aed7fa640 1 -- 172.21.15.33:0/1165735055 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f0af00cbca0 con 0x7f0ae80c1790 2024-03-29T11:39:43.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.915+0000 7f0aed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:43.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.915+0000 7f0aed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:43.919 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:43.919 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 10 librados: watch_flush enter 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 10 librados: watch_flush exit 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 10 client.4408.objecter successfully canceled tick 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 >> v1:172.21.15.153:6816/2737028074 conn(0x7f0adc06d9b0 legacy=0x7f0adc06fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 10 monclient: shutdown 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 >> v1:172.21.15.153:6789/0 conn(0x7f0ae80c1790 legacy=0x7f0ae815c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 shutdown_connections 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.922+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 >> 172.21.15.33:0/1165735055 conn(0x7f0ae8005c00 msgr2=0x7f0ae80c4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.923+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 shutdown_connections 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.923+0000 7f0af5b6d640 1 -- 172.21.15.33:0/1165735055 wait complete. 2024-03-29T11:39:43.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:43.923+0000 7f0af5b6d640 1 librados: shutdown 2024-03-29T11:39:43.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:50.748 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.900+0000 7fbd21872640 10 monclient: get_monmap_and_config 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.900+0000 7fbd21872640 10 monclient: build_initial_monmap 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.900+0000 7fbd21872640 10 monclient: monmap: 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:50.900754+0000 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:50.900754+0000 2024-03-29T11:39:50.902 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:50.903 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:50.903 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:50.903 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:50.903 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:50.903 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:50.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.902+0000 7fbd21872640 1 Processor -- start 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.902+0000 7fbd21872640 1 -- start start 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.902+0000 7fbd21872640 10 monclient: init 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 10 monclient(hunting): picked mon.noname-b con 0x7fbd14034850 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1402a640 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 10 monclient(hunting): picked mon.noname-c con 0x7fbd140327d0 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1402c820 con 0x7fbd1402a640 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1402a120 con 0x7fbd140327d0 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd140c75b0 con 0x7fbd14034850 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:50.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.903+0000 7fbd21872640 10 monclient(hunting): authenticate will time out at 1147.167068s 2024-03-29T11:39:50.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.906+0000 7fbd20870640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fbd14034850 0x7fbd140c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:34377/0 (socket says 172.21.15.33:34377) 2024-03-29T11:39:50.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.906+0000 7fbd20870640 1 -- 172.21.15.33:0/2801158097 learned_addr learned my addr 172.21.15.33:0/2801158097 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:50.909 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.908+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724476714 0 0) 0x7fbd140c75b0 con 0x7fbd14034850 2024-03-29T11:39:50.909 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.908+0000 7fbd1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:50.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.908+0000 7fbd1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:50.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.908+0000 7fbd1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:50.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.908+0000 7fbd1b7fe640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:39:50.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.908+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd0c003520 con 0x7fbd14034850 2024-03-29T11:39:50.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.910+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402315093 0 0) 0x7fbd1402a120 con 0x7fbd140327d0 2024-03-29T11:39:50.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.910+0000 7fbd1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:50.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.910+0000 7fbd1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:50.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.910+0000 7fbd1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:50.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.910+0000 7fbd1b7fe640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:39:50.912 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.910+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd140c75b0 con 0x7fbd140327d0 2024-03-29T11:39:50.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620142281 0 0) 0x7fbd0c003520 con 0x7fbd14034850 2024-03-29T11:39:50.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 >> v1:172.21.15.153:6789/0 conn(0x7fbd140327d0 legacy=0x7fbd140cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 >> v1:172.21.15.33:6789/0 conn(0x7fbd1402a640 legacy=0x7fbd140320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:50.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:50.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:50.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:50.914 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14033110 con 0x7fbd14034850 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbd08001690 con 0x7fbd14034850 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.911+0000 7fbd1b7fe640 10 monclient: dump: 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:50.915 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:50.916 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:50.916 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:50.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.912+0000 7fbd21872640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:39:50.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.912+0000 7fbd21872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:50.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.913+0000 7fbd1b7fe640 1 -- 172.21.15.33:0/2801158097 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd08001860 con 0x7fbd14034850 2024-03-29T11:39:50.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.913+0000 7fbd1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:50.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.913+0000 7fbd21872640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:50.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.913+0000 7fbd21872640 4 set_mon_vals no callback set 2024-03-29T11:39:50.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.914+0000 7fbd21872640 10 monclient: shutdown 2024-03-29T11:39:50.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.914+0000 7fbd21872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:50.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.914+0000 7fbd21872640 1 -- 172.21.15.33:0/2801158097 >> v1:172.21.15.33:6790/0 conn(0x7fbd14034850 legacy=0x7fbd140c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.914+0000 7fbd21872640 1 -- 172.21.15.33:0/2801158097 shutdown_connections 2024-03-29T11:39:50.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.914+0000 7fbd21872640 1 -- 172.21.15.33:0/2801158097 >> 172.21.15.33:0/2801158097 conn(0x7fbd140058f0 msgr2=0x7fbd14005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:50.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.917+0000 7fbd21872640 1 -- 172.21.15.33:0/2801158097 shutdown_connections 2024-03-29T11:39:50.919 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.917+0000 7fbd21872640 1 -- 172.21.15.33:0/2801158097 wait complete. 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.918+0000 7fbd21872640 10 monclient: build_initial_monmap 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.918+0000 7fbd21872640 10 monclient: monmap: 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.918+0000 7fbd21872640 1 librados: starting msgr at 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.918+0000 7fbd21872640 1 librados: starting objecter 2024-03-29T11:39:50.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.918+0000 7fbd21872640 1 Processor -- start 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.919+0000 7fbd21872640 1 -- start start 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.919+0000 7fbd21872640 1 librados: setting wanted keys 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.919+0000 7fbd21872640 1 librados: calling monclient init 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.919+0000 7fbd21872640 10 monclient: init 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 10 monclient(hunting): picked mon.noname-c con 0x7fbd1402a640 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 10 monclient(hunting): picked mon.noname-a con 0x7fbd140327d0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 10 monclient(hunting): picked mon.noname-b con 0x7fbd140c3ef0 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd140c3320 con 0x7fbd140327d0 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd140c2e00 con 0x7fbd140c3ef0 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd14161c20 con 0x7fbd1402a640 2024-03-29T11:39:50.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:50.925 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.920+0000 7fbd21872640 10 monclient(hunting): authenticate will time out at 1147.184364s 2024-03-29T11:39:50.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.925+0000 7fbd1bfff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fbd140327d0 0x7fbd14161510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:50325/0 (socket says 172.21.15.33:50325) 2024-03-29T11:39:50.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.925+0000 7fbd1bfff640 1 -- 172.21.15.33:0/1914912237 learned_addr learned my addr 172.21.15.33:0/1914912237 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:50.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.926+0000 7fbd197fa640 10 client.?.objecter ms_handle_connect 0x7fbd140327d0 2024-03-29T11:39:50.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.926+0000 7fbd197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:50.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787331311 0 0) 0x7fbd140c3320 con 0x7fbd140327d0 2024-03-29T11:39:50.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:50.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:50.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:50.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:39:50.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd00004eb0 con 0x7fbd140327d0 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3998099105 0 0) 0x7fbd00004eb0 con 0x7fbd140327d0 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd00005300 con 0x7fbd140327d0 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbd10001920 con 0x7fbd140327d0 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): dump: 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:50.930 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.928+0000 7fbd197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 529056509 0 0) 0x7fbd00005300 con 0x7fbd140327d0 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 >> v1:172.21.15.33:6790/0 conn(0x7fbd1402a640 legacy=0x7fbd140c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 >> v1:172.21.15.153:6789/0 conn(0x7fbd140c3ef0 legacy=0x7fbd140c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 1 monclient: found mon.a 2024-03-29T11:39:50.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:50.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:50.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14161db0 con 0x7fbd140327d0 2024-03-29T11:39:50.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd10001cc0 con 0x7fbd140327d0 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbd10003a40 con 0x7fbd140327d0 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd197fa640 10 monclient: dump: 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:50.933 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd1affd640 4 set_mon_vals no callback set 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 10 monclient: _renew_subs 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd14161fa0 con 0x7fbd140327d0 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 10 monclient: _renew_subs 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd140bfb70 con 0x7fbd140327d0 2024-03-29T11:39:50.934 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.929+0000 7fbd21872640 1 librados: init done 2024-03-29T11:39:50.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.934+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fbd10003c20 con 0x7fbd140327d0 2024-03-29T11:39:50.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.935+0000 7fbd21872640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:50.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.935+0000 7fbd21872640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:50.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.935+0000 7fbd21872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:50.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.935+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd14004cd0 con 0x7fbd140327d0 2024-03-29T11:39:50.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.936+0000 7fbd20870640 10 monclient: get_auth_request con 0x7fbd0006e060 auth_method 0 2024-03-29T11:39:50.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.936+0000 7fbd197fa640 10 client.4401.objecter ms_handle_connect 0x7fbd0006e060 2024-03-29T11:39:50.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.937+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fbd100545b0 con 0x7fbd140327d0 2024-03-29T11:39:50.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.937+0000 7fbd197fa640 10 client.4401.objecter ms_dispatch 0x7fbd1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:50.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.937+0000 7fbd197fa640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:50.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.937+0000 7fbd197fa640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:50.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.937+0000 7fbd197fa640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:39:50.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.944+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd10050850 con 0x7fbd140327d0 2024-03-29T11:39:50.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.944+0000 7fbd197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:50.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:50.944+0000 7fbd197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:51.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.272+0000 7fbd21872640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:51.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.272+0000 7fbd21872640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:51.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.272+0000 7fbd21872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:39:51.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.272+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd140c2f90 con 0x7fbd140327d0 2024-03-29T11:39:51.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.274+0000 7fbd197fa640 1 -- 172.21.15.33:0/1914912237 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fbd10050850 con 0x7fbd140327d0 2024-03-29T11:39:51.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.274+0000 7fbd197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:51.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.274+0000 7fbd197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:51.276 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:51.276 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:51.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.280+0000 7fbd21872640 10 librados: watch_flush enter 2024-03-29T11:39:51.286 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.285+0000 7fbd21872640 10 librados: watch_flush exit 2024-03-29T11:39:51.286 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.285+0000 7fbd21872640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:39:51.286 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.285+0000 7fbd21872640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:39:51.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.286+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 >> v1:172.21.15.153:6816/2737028074 conn(0x7fbd0006e060 legacy=0x7fbd00070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:51.288 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.287+0000 7fbd21872640 10 monclient: shutdown 2024-03-29T11:39:51.288 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.287+0000 7fbd21872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:51.288 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.287+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 >> v1:172.21.15.33:6789/0 conn(0x7fbd140327d0 legacy=0x7fbd14161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:51.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.288+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 shutdown_connections 2024-03-29T11:39:51.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.288+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 >> 172.21.15.33:0/1914912237 conn(0x7fbd140058f0 msgr2=0x7fbd14005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:51.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.288+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 shutdown_connections 2024-03-29T11:39:51.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.288+0000 7fbd21872640 1 -- 172.21.15.33:0/1914912237 wait complete. 2024-03-29T11:39:51.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:51.288+0000 7fbd21872640 1 librados: shutdown 2024-03-29T11:39:51.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:58.113 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.247+0000 7f81b320f640 10 monclient: get_monmap_and_config 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.247+0000 7f81b320f640 10 monclient: build_initial_monmap 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.247+0000 7f81b320f640 10 monclient: monmap: 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:39:58.248131+0000 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:39:58.248131+0000 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:58.249 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:39:58.250 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:39:58.250 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:58.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.249+0000 7f81b320f640 1 Processor -- start 2024-03-29T11:39:58.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.249+0000 7f81b320f640 1 -- start start 2024-03-29T11:39:58.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.249+0000 7f81b320f640 10 monclient: init 2024-03-29T11:39:58.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.250+0000 7f81b320f640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:58.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.250+0000 7f81b320f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:58.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.250+0000 7f81b320f640 10 monclient(hunting): picked mon.noname-c con 0x7f81ac1457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:58.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b320f640 10 monclient(hunting): picked mon.noname-b con 0x7f81ac13f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:58.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b0fab640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f81ac1457f0 0x7f81ac13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:56270/0 (socket says 172.21.15.33:56270) 2024-03-29T11:39:58.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b0fab640 1 -- 172.21.15.33:0/3367107221 learned_addr learned my addr 172.21.15.33:0/3367107221 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:58.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b320f640 10 monclient(hunting): picked mon.noname-a con 0x7f81ac0a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:58.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0aaef0 con 0x7f81ac0a9d70 2024-03-29T11:39:58.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac145bd0 con 0x7f81ac1457f0 2024-03-29T11:39:58.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac145eb0 con 0x7f81ac13f650 2024-03-29T11:39:58.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b320f640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:58.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.251+0000 7f81b320f640 10 monclient(hunting): authenticate will time out at 1154.515380s 2024-03-29T11:39:58.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938210999 0 0) 0x7f81ac145bd0 con 0x7f81ac1457f0 2024-03-29T11:39:58.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:39:58.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f819c003520 con 0x7f81ac1457f0 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500234956 0 0) 0x7f81ac145eb0 con 0x7f81ac13f650 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.253+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f819c003840 con 0x7f81ac13f650 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 226728311 0 0) 0x7f819c003520 con 0x7f81ac1457f0 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 >> v1:172.21.15.33:6790/0 conn(0x7f81ac13f650 legacy=0x7f81ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 >> v1:172.21.15.33:6789/0 conn(0x7f81ac0a9d70 legacy=0x7f81ac14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:58.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:58.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 1 monclient: found mon.noname-c 2024-03-29T11:39:58.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:58.256 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac146340 con 0x7f81ac1457f0 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f81a0001670 con 0x7f81ac1457f0 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.254+0000 7f81ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81ab7fe640 10 monclient: dump: 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:58.257 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81b320f640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81b320f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a0001850 con 0x7f81ac1457f0 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81ab7fe640 1 -- 172.21.15.33:0/3367107221 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f81a0001c60 con 0x7f81ac1457f0 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81ab7fe640 10 monclient: dump: 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.258 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81b320f640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:58.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.255+0000 7f81b320f640 4 set_mon_vals no callback set 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.256+0000 7f81b320f640 10 monclient: shutdown 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.256+0000 7f81b320f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.256+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 >> v1:172.21.15.153:6789/0 conn(0x7f81ac1457f0 legacy=0x7f81ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.256+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 shutdown_connections 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.256+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 >> 172.21.15.33:0/3367107221 conn(0x7f81ac066a00 msgr2=0x7f81ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.257+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 shutdown_connections 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.257+0000 7f81b320f640 1 -- 172.21.15.33:0/3367107221 wait complete. 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.257+0000 7f81b320f640 10 monclient: build_initial_monmap 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.257+0000 7f81b320f640 10 monclient: monmap: 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:58.266 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.258+0000 7f81b320f640 1 librados: starting msgr at 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.258+0000 7f81b320f640 1 librados: starting objecter 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.258+0000 7f81b320f640 1 Processor -- start 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.258+0000 7f81b320f640 1 -- start start 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.258+0000 7f81b320f640 1 librados: setting wanted keys 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.258+0000 7f81b320f640 1 librados: calling monclient init 2024-03-29T11:39:58.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.258+0000 7f81b320f640 10 monclient: init 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.259+0000 7f81b320f640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.259+0000 7f81b320f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.259+0000 7f81b320f640 10 monclient(hunting): picked mon.noname-b con 0x7f81ac0a9d70 addr v1:172.21.15.153:6789/0 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.260+0000 7f81b320f640 10 monclient(hunting): picked mon.noname-c con 0x7f81ac13f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.260+0000 7f81b0fab640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f81ac0a9d70 0x7f81ac1403f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:53694/0 (socket says 172.21.15.33:53694) 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.260+0000 7f81abfff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f81ac13f650 0x7f81ac136b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:53852/0 (socket says 172.21.15.33:53852) 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.260+0000 7f81b320f640 10 monclient(hunting): picked mon.noname-a con 0x7f81ac1457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.260+0000 7f81b0fab640 1 -- 172.21.15.33:0/2839213865 learned_addr learned my addr 172.21.15.33:0/2839213865 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.260+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac13ff60 con 0x7f81ac1457f0 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac137990 con 0x7f81ac0a9d70 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0ac5d0 con 0x7f81ac13f650 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81b320f640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81b320f640 10 monclient(hunting): authenticate will time out at 1154.524928s 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 client.?.objecter ms_handle_connect 0x7f81ac13f650 2024-03-29T11:39:58.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392495218 0 0) 0x7f81ac0ac5d0 con 0x7f81ac13f650 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8190004eb0 con 0x7f81ac13f650 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 client.?.objecter ms_handle_connect 0x7f81ac0a9d70 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723151134 0 0) 0x7f81ac137990 con 0x7f81ac0a9d70 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.261+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac0ac5d0 con 0x7f81ac0a9d70 2024-03-29T11:39:58.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 720732424 0 0) 0x7f8190004eb0 con 0x7f81ac13f650 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81ac137990 con 0x7f81ac13f650 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804572668 0 0) 0x7f81ac0ac5d0 con 0x7f81ac0a9d70 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8190004eb0 con 0x7f81ac0a9d70 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8198001950 con 0x7f81ac13f650 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 10 monclient(hunting): dump: 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:58.270 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f81a0001b60 con 0x7f81ac0a9d70 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.262+0000 7f81a97fa640 10 monclient(hunting): dump: 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:58.271 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 195919585 0 0) 0x7f81ac137990 con 0x7f81ac13f650 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 >> v1:172.21.15.153:6789/0 conn(0x7f81ac0a9d70 legacy=0x7f81ac1403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 >> v1:172.21.15.33:6789/0 conn(0x7f81ac1457f0 legacy=0x7f81ac137280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 1 monclient: found mon.c 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac0ac8b0 con 0x7f81ac13f650 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81b320f640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81b320f640 10 monclient: _renew_subs 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81b320f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:58.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac137c10 con 0x7f81ac13f650 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81b320f640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8198003610 con 0x7f81ac13f650 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8198003a20 con 0x7f81ac13f650 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.263+0000 7f81a97fa640 10 monclient: dump: 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:39:58.273 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:39:58.274 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:39:58.274 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:39:58.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.264+0000 7f81b320f640 10 monclient: _renew_subs 2024-03-29T11:39:58.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.264+0000 7f81aaffd640 4 set_mon_vals no callback set 2024-03-29T11:39:58.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.264+0000 7f81b320f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:58.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.264+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac0ab600 con 0x7f81ac13f650 2024-03-29T11:39:58.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.264+0000 7f81b320f640 1 librados: init done 2024-03-29T11:39:58.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.265+0000 7f81b320f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:58.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.265+0000 7f81b320f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:58.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.265+0000 7f81b320f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:58.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.265+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ac140110 con 0x7f81ac13f650 2024-03-29T11:39:58.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.266+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f8198003b80 con 0x7f81ac13f650 2024-03-29T11:39:58.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.266+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f81980849a0 con 0x7f81ac13f650 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.266+0000 7f81a97fa640 10 client.4457.objecter ms_dispatch 0x7f81ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.266+0000 7f81a97fa640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.266+0000 7f81a97fa640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.266+0000 7f81a97fa640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.266+0000 7f81b0fab640 10 monclient: get_auth_request con 0x7f819006de90 auth_method 0 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.267+0000 7f81a97fa640 10 client.4457.objecter ms_handle_connect 0x7f819006de90 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.275+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81ac140110 con 0x7f81ac13f650 2024-03-29T11:39:58.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.275+0000 7f81a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:58.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.275+0000 7f81a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:58.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.587+0000 7f81b320f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:58.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.588+0000 7f81b320f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:58.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.588+0000 7f81b320f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:39:58.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.588+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81ac140110 con 0x7f81ac13f650 2024-03-29T11:39:58.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.589+0000 7f81a97fa640 1 -- 172.21.15.33:0/2839213865 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f81ac140110 con 0x7f81ac13f650 2024-03-29T11:39:58.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.589+0000 7f81a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:58.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.589+0000 7f81a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:58.590 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:39:58.590 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:58.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 10 librados: watch_flush enter 2024-03-29T11:39:58.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 10 librados: watch_flush exit 2024-03-29T11:39:58.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:39:58.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:39:58.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 >> v1:172.21.15.153:6816/2737028074 conn(0x7f819006de90 legacy=0x7f8190070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 10 monclient: shutdown 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 >> v1:172.21.15.33:6790/0 conn(0x7f81ac13f650 legacy=0x7f81ac136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 shutdown_connections 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.592+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 >> 172.21.15.33:0/2839213865 conn(0x7f81ac066a00 msgr2=0x7f81ac13aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.593+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 shutdown_connections 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.593+0000 7f81b320f640 1 -- 172.21.15.33:0/2839213865 wait complete. 2024-03-29T11:39:58.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:39:58.593+0000 7f81b320f640 1 librados: shutdown 2024-03-29T11:39:58.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:05.415 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:05.580 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.578+0000 7fa8956b3640 10 monclient: get_monmap_and_config 2024-03-29T11:40:05.580 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.578+0000 7fa8956b3640 10 monclient: build_initial_monmap 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.578+0000 7fa8956b3640 10 monclient: monmap: 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:05.579855+0000 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:05.579855+0000 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:05.581 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.579+0000 7fa8956b3640 1 Processor -- start 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.579+0000 7fa8956b3640 1 -- start start 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.579+0000 7fa8956b3640 10 monclient: init 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 10 monclient(hunting): picked mon.noname-b con 0x7fa8900b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 10 monclient(hunting): picked mon.noname-a con 0x7fa89014a700 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa8900ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:05.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900aba70 con 0x7fa89014a700 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900aecb0 con 0x7fa8900ac5a0 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900af150 con 0x7fa8900b0770 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa8956b3640 10 monclient(hunting): authenticate will time out at 1161.845190s 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa88f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa8900ac5a0 0x7fa890151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:33562/0 (socket says 172.21.15.33:33562) 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.580+0000 7fa88f7fe640 1 -- 172.21.15.33:0/729555171 learned_addr learned my addr 172.21.15.33:0/729555171 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159002825 0 0) 0x7fa8900aecb0 con 0x7fa8900ac5a0 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:40:05.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa878003520 con 0x7fa8900ac5a0 2024-03-29T11:40:05.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839352481 0 0) 0x7fa878003520 con 0x7fa8900ac5a0 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 >> v1:172.21.15.33:6790/0 conn(0x7fa8900b0770 legacy=0x7fa890149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 >> v1:172.21.15.33:6789/0 conn(0x7fa89014a700 legacy=0x7fa8900abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 monclient: found mon.noname-c 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8900ac9d0 con 0x7fa8900ac5a0 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa8956b3640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa884001670 con 0x7fa8900ac5a0 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa8956b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.581+0000 7fa88dffb640 10 monclient: dump: 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa8956b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa884001840 con 0x7fa8900ac5a0 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa88dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa8956b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:05.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa8956b3640 4 set_mon_vals no callback set 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa88dffb640 1 -- 172.21.15.33:0/729555171 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa884001c50 con 0x7fa8900ac5a0 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa88dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa88dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.582+0000 7fa88dffb640 10 monclient: dump: 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.583+0000 7fa8956b3640 10 monclient: shutdown 2024-03-29T11:40:05.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.583+0000 7fa8956b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.583+0000 7fa8956b3640 1 -- 172.21.15.33:0/729555171 >> v1:172.21.15.153:6789/0 conn(0x7fa8900ac5a0 legacy=0x7fa890151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.583+0000 7fa8956b3640 1 -- 172.21.15.33:0/729555171 shutdown_connections 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.583+0000 7fa8956b3640 1 -- 172.21.15.33:0/729555171 >> 172.21.15.33:0/729555171 conn(0x7fa89006b660 msgr2=0x7fa89006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.583+0000 7fa8956b3640 1 -- 172.21.15.33:0/729555171 shutdown_connections 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.583+0000 7fa8956b3640 1 -- 172.21.15.33:0/729555171 wait complete. 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.584+0000 7fa8956b3640 10 monclient: build_initial_monmap 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.584+0000 7fa8956b3640 10 monclient: monmap: 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:05.590 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:05.591 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:05.591 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:05.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.584+0000 7fa8956b3640 1 librados: starting msgr at 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.584+0000 7fa8956b3640 1 librados: starting objecter 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.585+0000 7fa8956b3640 1 Processor -- start 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.585+0000 7fa8956b3640 1 -- start start 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.585+0000 7fa8956b3640 1 librados: setting wanted keys 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.585+0000 7fa8956b3640 1 librados: calling monclient init 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.585+0000 7fa8956b3640 10 monclient: init 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa8956b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa8956b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa8956b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa8900ac5a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa8956b3640 10 monclient(hunting): picked mon.noname-b con 0x7fa8900b0770 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa8956b3640 10 monclient(hunting): picked mon.noname-a con 0x7fa89014a700 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:05.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa8956b3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89012a5d0 con 0x7fa89014a700 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa88e7fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa8900b0770 0x7fa890128a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:34118/0 (socket says 172.21.15.33:34118) 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa88e7fc640 1 -- 172.21.15.33:0/567968527 learned_addr learned my addr 172.21.15.33:0/567968527 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.586+0000 7fa8956b3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa890128550 con 0x7fa8900b0770 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8901255e0 con 0x7fa8900ac5a0 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8956b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8956b3640 10 monclient(hunting): authenticate will time out at 1161.851763s 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 client.?.objecter ms_handle_connect 0x7fa8900b0770 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 client.?.objecter ms_handle_connect 0x7fa8900ac5a0 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213435981 0 0) 0x7fa890128550 con 0x7fa8900b0770 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:40:05.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.587+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87c004eb0 con 0x7fa8900b0770 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088189543 0 0) 0x7fa87c004eb0 con 0x7fa8900b0770 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa890128550 con 0x7fa8900b0770 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 client.?.objecter ms_handle_connect 0x7fa89014a700 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa880001990 con 0x7fa8900b0770 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): dump: 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:05.598 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721037587 0 0) 0x7fa8901255e0 con 0x7fa8900ac5a0 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87c004eb0 con 0x7fa8900ac5a0 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152815302 0 0) 0x7fa89012a5d0 con 0x7fa89014a700 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.588+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8901255e0 con 0x7fa89014a700 2024-03-29T11:40:05.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 107977454 0 0) 0x7fa890128550 con 0x7fa8900b0770 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 >> v1:172.21.15.33:6790/0 conn(0x7fa8900ac5a0 legacy=0x7fa890147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 >> v1:172.21.15.33:6789/0 conn(0x7fa89014a700 legacy=0x7fa8901e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 1 monclient: found mon.b 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8901478a0 con 0x7fa8900b0770 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 10 monclient: _renew_subs 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8901dfa60 con 0x7fa8900b0770 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 10 monclient: _renew_subs 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:05.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa890128150 con 0x7fa8900b0770 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8956b3640 1 librados: init done 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa880001c50 con 0x7fa8900b0770 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa880003a60 con 0x7fa8900b0770 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa8777fe640 10 monclient: dump: 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:05.601 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.589+0000 7fa88cff9640 4 set_mon_vals no callback set 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.592+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fa880003bc0 con 0x7fa8900b0770 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.592+0000 7fa8956b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8956b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8956b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fa880084aa0 con 0x7fa8900b0770 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8777fe640 10 client.4423.objecter ms_dispatch 0x7fa890000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8777fe640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8777fe640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa870005080 con 0x7fa8900b0770 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa8777fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:40:05.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.593+0000 7fa88effd640 10 monclient: get_auth_request con 0x7fa87c06deb0 auth_method 0 2024-03-29T11:40:05.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.594+0000 7fa8777fe640 10 client.4423.objecter ms_handle_connect 0x7fa87c06deb0 2024-03-29T11:40:05.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.600+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa880050380 con 0x7fa8900b0770 2024-03-29T11:40:05.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.600+0000 7fa8777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:05.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:05.600+0000 7fa8777fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:06.023 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.021+0000 7fa8956b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:06.023 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.021+0000 7fa8956b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:06.024 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.021+0000 7fa8956b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:06.024 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.021+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa870005330 con 0x7fa8900b0770 2024-03-29T11:40:06.026 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:06.026 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:06.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.023+0000 7fa8777fe640 1 -- 172.21.15.33:0/567968527 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fa8800522b0 con 0x7fa8900b0770 2024-03-29T11:40:06.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.023+0000 7fa8777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:06.027 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.023+0000 7fa8777fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:06.029 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 10 librados: watch_flush enter 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 10 librados: watch_flush exit 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 >> v1:172.21.15.153:6816/2737028074 conn(0x7fa87c06deb0 legacy=0x7fa87c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 10 monclient: shutdown 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 >> v1:172.21.15.153:6789/0 conn(0x7fa8900b0770 legacy=0x7fa890128a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:06.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 shutdown_connections 2024-03-29T11:40:06.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 >> 172.21.15.33:0/567968527 conn(0x7fa89006b660 msgr2=0x7fa8900abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:06.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 shutdown_connections 2024-03-29T11:40:06.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 1 -- 172.21.15.33:0/567968527 wait complete. 2024-03-29T11:40:06.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:06.027+0000 7fa8956b3640 1 librados: shutdown 2024-03-29T11:40:06.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:12.852 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:13.023 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.021+0000 7f19b9c46640 10 monclient: get_monmap_and_config 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.021+0000 7f19b9c46640 10 monclient: build_initial_monmap 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.021+0000 7f19b9c46640 10 monclient: monmap: 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:13.022691+0000 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:13.022691+0000 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:13.024 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:13.025 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.023+0000 7f19b9c46640 1 Processor -- start 2024-03-29T11:40:13.025 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.023+0000 7f19b9c46640 1 -- start start 2024-03-29T11:40:13.026 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.023+0000 7f19b9c46640 10 monclient: init 2024-03-29T11:40:13.028 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.024+0000 7f19b9c46640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:13.028 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.024+0000 7f19b9c46640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:13.028 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.024+0000 7f19b9c46640 10 monclient(hunting): picked mon.noname-c con 0x7f19ac006890 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:13.029 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.024+0000 7f19b9c46640 10 monclient(hunting): picked mon.noname-a con 0x7f19ac006c70 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:13.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.027+0000 7f19b9c46640 10 monclient(hunting): picked mon.noname-b con 0x7f19ac02b4c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:13.033 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.031+0000 7f19b8c44640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f19ac006890 0x7f19ac0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:46315/0 (socket says 172.21.15.33:46315) 2024-03-29T11:40:13.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.031+0000 7f19b8c44640 1 -- 172.21.15.33:0/608212417 learned_addr learned my addr 172.21.15.33:0/608212417 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:13.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.031+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac02c830 con 0x7f19ac006c70 2024-03-29T11:40:13.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.031+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac02afa0 con 0x7f19ac006890 2024-03-29T11:40:13.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.031+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac02aba0 con 0x7f19ac02b4c0 2024-03-29T11:40:13.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.031+0000 7f19b9c46640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:13.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.031+0000 7f19b9c46640 10 monclient(hunting): authenticate will time out at 1169.296027s 2024-03-29T11:40:13.037 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943545589 0 0) 0x7f19ac02c830 con 0x7f19ac006c70 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199c003520 con 0x7f19ac006c70 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274472503 0 0) 0x7f19ac02afa0 con 0x7f19ac006890 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac02c830 con 0x7f19ac006890 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011832111 0 0) 0x7f19ac02aba0 con 0x7f19ac02b4c0 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.034+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac02afa0 con 0x7f19ac02b4c0 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177652232 0 0) 0x7f199c003520 con 0x7f19ac006c70 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 >> v1:172.21.15.33:6790/0 conn(0x7f19ac02b4c0 legacy=0x7f19ac0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 >> v1:172.21.15.153:6789/0 conn(0x7f19ac006890 legacy=0x7f19ac0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 1 monclient: found mon.noname-a 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac02ad30 con 0x7f19ac006c70 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:13.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f19a4001690 con 0x7f19ac006c70 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19ab7fe640 10 monclient: dump: 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19b9c46640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:40:13.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.035+0000 7f19b9c46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.038+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19a40017f0 con 0x7f19ac006c70 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.038+0000 7f19ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.038+0000 7f19ab7fe640 1 -- 172.21.15.33:0/608212417 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f19a4001b80 con 0x7f19ac006c70 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.038+0000 7f19b9c46640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.038+0000 7f19b9c46640 4 set_mon_vals no callback set 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.039+0000 7f19ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.039+0000 7f19ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.039+0000 7f19ab7fe640 10 monclient: dump: 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.042 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.039+0000 7f19b9c46640 10 monclient: shutdown 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.039+0000 7f19b9c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.039+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 >> v1:172.21.15.33:6789/0 conn(0x7f19ac006c70 legacy=0x7f19ac0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.040+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 shutdown_connections 2024-03-29T11:40:13.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.040+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 >> 172.21.15.33:0/608212417 conn(0x7f19ac0058f0 msgr2=0x7f19ac005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:13.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.044+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 shutdown_connections 2024-03-29T11:40:13.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.046+0000 7f19b9c46640 1 -- 172.21.15.33:0/608212417 wait complete. 2024-03-29T11:40:13.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.047+0000 7f19b9c46640 10 monclient: build_initial_monmap 2024-03-29T11:40:13.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.047+0000 7f19b9c46640 10 monclient: monmap: 2024-03-29T11:40:13.049 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:13.049 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:13.049 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:13.050 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:13.050 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:13.050 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:13.050 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:13.050 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:13.050 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:13.050 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:13.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.048+0000 7f19b9c46640 1 librados: starting msgr at 2024-03-29T11:40:13.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.048+0000 7f19b9c46640 1 librados: starting objecter 2024-03-29T11:40:13.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.048+0000 7f19b9c46640 1 Processor -- start 2024-03-29T11:40:13.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.048+0000 7f19b9c46640 1 -- start start 2024-03-29T11:40:13.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.048+0000 7f19b9c46640 1 librados: setting wanted keys 2024-03-29T11:40:13.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.048+0000 7f19b9c46640 1 librados: calling monclient init 2024-03-29T11:40:13.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.048+0000 7f19b9c46640 10 monclient: init 2024-03-29T11:40:13.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.050+0000 7f19b9c46640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:13.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.050+0000 7f19b9c46640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:13.055 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.050+0000 7f19b9c46640 10 monclient(hunting): picked mon.noname-a con 0x7f19ac006890 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:13.055 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.053+0000 7f19b9c46640 10 monclient(hunting): picked mon.noname-b con 0x7f19ac02b4c0 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:13.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.053+0000 7f19b9c46640 10 monclient(hunting): picked mon.noname-c con 0x7f19ac0310a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:13.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.057+0000 7f19b9c46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac15c8b0 con 0x7f19ac006890 2024-03-29T11:40:13.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.057+0000 7f19b9c46640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac15c580 con 0x7f19ac02b4c0 2024-03-29T11:40:13.060 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.057+0000 7f19b9c46640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac031b40 con 0x7f19ac0310a0 2024-03-29T11:40:13.060 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.057+0000 7f19b9c46640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:13.060 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.057+0000 7f19b9c46640 10 monclient(hunting): authenticate will time out at 1169.322060s 2024-03-29T11:40:13.060 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.057+0000 7f19b8c44640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f19ac006890 0x7f19ac15be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:45625/0 (socket says 172.21.15.33:45625) 2024-03-29T11:40:13.060 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.057+0000 7f19b8c44640 1 -- 172.21.15.33:0/1716041867 learned_addr learned my addr 172.21.15.33:0/1716041867 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:13.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.059+0000 7f19a97fa640 10 client.?.objecter ms_handle_connect 0x7f19ac0310a0 2024-03-29T11:40:13.061 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.059+0000 7f19a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:13.064 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 client.?.objecter ms_handle_connect 0x7f19ac006890 2024-03-29T11:40:13.064 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:13.064 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874593253 0 0) 0x7f19ac031b40 con 0x7f19ac0310a0 2024-03-29T11:40:13.064 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.064 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.064 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.064 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a0004eb0 con 0x7f19ac0310a0 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515808575 0 0) 0x7f19ac15c8b0 con 0x7f19ac006890 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.060+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac031b40 con 0x7f19ac006890 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 client.?.objecter ms_handle_connect 0x7f19ac02b4c0 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1426316248 0 0) 0x7f19a0004eb0 con 0x7f19ac0310a0 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19ac15c8b0 con 0x7f19ac0310a0 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f19b4068810 con 0x7f19ac0310a0 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.065 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): dump: 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014504876 0 0) 0x7f19ac031b40 con 0x7f19ac006890 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19a0004eb0 con 0x7f19ac006890 2024-03-29T11:40:13.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531197947 0 0) 0x7f19ac15c580 con 0x7f19ac02b4c0 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac031b40 con 0x7f19ac02b4c0 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f19b0001910 con 0x7f19ac006890 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient(hunting): dump: 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:13.067 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1144610961 0 0) 0x7f19ac15c8b0 con 0x7f19ac0310a0 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 >> v1:172.21.15.153:6789/0 conn(0x7f19ac02b4c0 legacy=0x7f19ac0c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 >> v1:172.21.15.33:6789/0 conn(0x7f19ac006890 legacy=0x7f19ac15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 monclient: found mon.c 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac1671f0 con 0x7f19ac0310a0 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:13.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19b9c46640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19b9c46640 10 monclient: _renew_subs 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19b9c46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac1641c0 con 0x7f19ac0310a0 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.061+0000 7f19b9c46640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19b9c46640 10 monclient: _renew_subs 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19b9c46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac164650 con 0x7f19ac0310a0 2024-03-29T11:40:13.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19b9c46640 1 librados: init done 2024-03-29T11:40:13.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b404f400 con 0x7f19ac0310a0 2024-03-29T11:40:13.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:13.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f19b4075f10 con 0x7f19ac0310a0 2024-03-29T11:40:13.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19aaffd640 4 set_mon_vals no callback set 2024-03-29T11:40:13.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:13.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.062+0000 7f19a97fa640 10 monclient: dump: 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.063+0000 7f19b9c46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.063+0000 7f19b9c46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.063+0000 7f19b9c46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:13.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.063+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1994005080 con 0x7f19ac0310a0 2024-03-29T11:40:13.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.064+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f19b4067bf0 con 0x7f19ac0310a0 2024-03-29T11:40:13.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.065+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f19b40fe1d0 con 0x7f19ac0310a0 2024-03-29T11:40:13.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.065+0000 7f19a97fa640 10 client.4466.objecter ms_dispatch 0x7f19b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:13.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.065+0000 7f19a97fa640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:13.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.065+0000 7f19a97fa640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:13.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.065+0000 7f19a97fa640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T11:40:13.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.068+0000 7f19b8c44640 10 monclient: get_auth_request con 0x7f19a006def0 auth_method 0 2024-03-29T11:40:13.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.071+0000 7f19a97fa640 10 client.4466.objecter ms_handle_connect 0x7f19a006def0 2024-03-29T11:40:13.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.075+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19b40c9ab0 con 0x7f19ac0310a0 2024-03-29T11:40:13.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.075+0000 7f19a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:13.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.075+0000 7f19a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:13.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.446+0000 7f19b9c46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:13.448 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.446+0000 7f19b9c46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:13.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.446+0000 7f19b9c46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:13.449 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.446+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1994005330 con 0x7f19ac0310a0 2024-03-29T11:40:13.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.448+0000 7f19a97fa640 1 -- 172.21.15.33:0/1716041867 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f19b40cb9e0 con 0x7f19ac0310a0 2024-03-29T11:40:13.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.448+0000 7f19a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:13.450 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.448+0000 7f19a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:13.450 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:13.450 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:13.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.451+0000 7f19b9c46640 10 librados: watch_flush enter 2024-03-29T11:40:13.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.451+0000 7f19b9c46640 10 librados: watch_flush exit 2024-03-29T11:40:13.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.451+0000 7f19b9c46640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T11:40:13.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.451+0000 7f19b9c46640 10 client.4466.objecter successfully canceled tick 2024-03-29T11:40:13.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.452+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 >> v1:172.21.15.153:6816/2737028074 conn(0x7f19a006def0 legacy=0x7f19a00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.454 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.452+0000 7f19b9c46640 10 monclient: shutdown 2024-03-29T11:40:13.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.452+0000 7f19b9c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:13.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.452+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 >> v1:172.21.15.33:6790/0 conn(0x7f19ac0310a0 legacy=0x7f19ac0c2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:13.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.452+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 shutdown_connections 2024-03-29T11:40:13.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.452+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 >> 172.21.15.33:0/1716041867 conn(0x7f19ac0058f0 msgr2=0x7f19ac038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:13.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.452+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 shutdown_connections 2024-03-29T11:40:13.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.453+0000 7f19b9c46640 1 -- 172.21.15.33:0/1716041867 wait complete. 2024-03-29T11:40:13.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:13.453+0000 7f19b9c46640 1 librados: shutdown 2024-03-29T11:40:13.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:20.277 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:20.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.408+0000 7f83f7ec9640 10 monclient: get_monmap_and_config 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.408+0000 7f83f7ec9640 10 monclient: build_initial_monmap 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.408+0000 7f83f7ec9640 10 monclient: monmap: 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:20.410022+0000 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:20.410022+0000 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:20.411 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:20.412 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:20.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.409+0000 7f83f7ec9640 1 Processor -- start 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.409+0000 7f83f7ec9640 1 -- start start 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.409+0000 7f83f7ec9640 10 monclient: init 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f83f00a9750 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f83f013e1e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f83f0071c30 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00ab300 con 0x7f83f0071c30 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00ac2f0 con 0x7f83f00a9750 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00ad8d0 con 0x7f83f013e1e0 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:20.414 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f7ec9640 10 monclient(hunting): authenticate will time out at 1176.675056s 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f5c65640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f83f00a9750 0x7f83f013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:32954/0 (socket says 172.21.15.33:32954) 2024-03-29T11:40:20.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.410+0000 7f83f5c65640 1 -- 172.21.15.33:0/1120387917 learned_addr learned my addr 172.21.15.33:0/1120387917 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902185900 0 0) 0x7f83f00ac2f0 con 0x7f83f00a9750 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83dc003520 con 0x7f83f00a9750 2024-03-29T11:40:20.417 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285046643 0 0) 0x7f83f00ad8d0 con 0x7f83f013e1e0 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f00ac2f0 con 0x7f83f013e1e0 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926645556 0 0) 0x7f83f00ac2f0 con 0x7f83f013e1e0 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 >> v1:172.21.15.153:6789/0 conn(0x7f83f00a9750 legacy=0x7f83f013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.412+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 >> v1:172.21.15.33:6789/0 conn(0x7f83f0071c30 legacy=0x7f83f00ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 10 monclient: _finish_hunting 0 2024-03-29T11:40:20.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 1 monclient: found mon.noname-b 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f00ada60 con 0x7f83f013e1e0 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 10 monclient: _finish_auth 0 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 10 monclient: _check_auth_tickets 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f83e0001630 con 0x7f83f013e1e0 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f4c63640 10 monclient: dump: 2024-03-29T11:40:20.419 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f7ec9640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:40:20.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.413+0000 7f83f7ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83e0001880 con 0x7f83f013e1e0 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f4c63640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f4c63640 1 -- 172.21.15.33:0/1120387917 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f83e0001c10 con 0x7f83f013e1e0 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f4c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f4c63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f4c63640 10 monclient: dump: 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.421 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f7ec9640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:20.422 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.414+0000 7f83f7ec9640 4 set_mon_vals no callback set 2024-03-29T11:40:20.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.415+0000 7f83f7ec9640 10 monclient: shutdown 2024-03-29T11:40:20.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.415+0000 7f83f7ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:20.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.415+0000 7f83f7ec9640 1 -- 172.21.15.33:0/1120387917 >> v1:172.21.15.33:6790/0 conn(0x7f83f013e1e0 legacy=0x7f83f00ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.415+0000 7f83f7ec9640 1 -- 172.21.15.33:0/1120387917 shutdown_connections 2024-03-29T11:40:20.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.415+0000 7f83f7ec9640 1 -- 172.21.15.33:0/1120387917 >> 172.21.15.33:0/1120387917 conn(0x7f83f006b660 msgr2=0x7f83f006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:20.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.415+0000 7f83f7ec9640 1 -- 172.21.15.33:0/1120387917 shutdown_connections 2024-03-29T11:40:20.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.415+0000 7f83f7ec9640 1 -- 172.21.15.33:0/1120387917 wait complete. 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.416+0000 7f83f7ec9640 10 monclient: build_initial_monmap 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.416+0000 7f83f7ec9640 10 monclient: monmap: 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:20.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.417+0000 7f83f7ec9640 1 librados: starting msgr at 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.417+0000 7f83f7ec9640 1 librados: starting objecter 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.417+0000 7f83f7ec9640 1 Processor -- start 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.417+0000 7f83f7ec9640 1 -- start start 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.417+0000 7f83f7ec9640 1 librados: setting wanted keys 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.417+0000 7f83f7ec9640 1 librados: calling monclient init 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.417+0000 7f83f7ec9640 10 monclient: init 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f83f0071c30 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f83f00a9750 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f83f013e1e0 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00b27f0 con 0x7f83f00a9750 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f01d6600 con 0x7f83f013e1e0 2024-03-29T11:40:20.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f0137350 con 0x7f83f0071c30 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f7ec9640 10 monclient(hunting): authenticate will time out at 1176.683224s 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f5c65640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f83f0071c30 0x7f83f00b1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:43446/0 (socket says 172.21.15.33:43446) 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.418+0000 7f83f5c65640 1 -- 172.21.15.33:0/3118476818 learned_addr learned my addr 172.21.15.33:0/3118476818 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 client.?.objecter ms_handle_connect 0x7f83f0071c30 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386288124 0 0) 0x7f83f0137350 con 0x7f83f0071c30 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d4004eb0 con 0x7f83f0071c30 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 client.?.objecter ms_handle_connect 0x7f83f013e1e0 2024-03-29T11:40:20.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567892028 0 0) 0x7f83f01d6600 con 0x7f83f013e1e0 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f0137350 con 0x7f83f013e1e0 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632992635 0 0) 0x7f83d4004eb0 con 0x7f83f0071c30 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83f01d6600 con 0x7f83f0071c30 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f83ec001850 con 0x7f83f0071c30 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.419+0000 7f83e67fc640 10 monclient(hunting): dump: 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.433 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.420+0000 7f83e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.420+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3602825905 0 0) 0x7f83f01d6600 con 0x7f83f0071c30 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.420+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 >> v1:172.21.15.153:6789/0 conn(0x7f83f013e1e0 legacy=0x7f83f01389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.420+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 >> v1:172.21.15.33:6789/0 conn(0x7f83f00a9750 legacy=0x7f83f01382a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:20.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 1 monclient: found mon.c 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f0139e50 con 0x7f83f0071c30 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83f7ec9640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.421+0000 7f83f7ec9640 10 monclient: _renew_subs 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83f7ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f00b18a0 con 0x7f83f0071c30 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83f7ec9640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83f7ec9640 10 monclient: _renew_subs 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83ec001d00 con 0x7f83f0071c30 2024-03-29T11:40:20.435 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83f7ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f01392a0 con 0x7f83f0071c30 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f83ec004980 con 0x7f83f0071c30 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83e67fc640 10 monclient: dump: 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:20.436 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83e7fff640 4 set_mon_vals no callback set 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.422+0000 7f83f7ec9640 1 librados: init done 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.424+0000 7f83f7ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.424+0000 7f83f7ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.424+0000 7f83f7ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.424+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f00a9b30 con 0x7f83f0071c30 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.424+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f83ec004ae0 con 0x7f83f0071c30 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.425+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f83ec084a80 con 0x7f83f0071c30 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.425+0000 7f83e67fc640 10 client.4472.objecter ms_dispatch 0x7f83f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.425+0000 7f83e67fc640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.425+0000 7f83e67fc640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.425+0000 7f83e67fc640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.425+0000 7f83f5464640 10 monclient: get_auth_request con 0x7f83d406dcb0 auth_method 0 2024-03-29T11:40:20.437 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.425+0000 7f83e67fc640 10 client.4472.objecter ms_handle_connect 0x7f83d406dcb0 2024-03-29T11:40:20.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.432+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83ec050190 con 0x7f83f0071c30 2024-03-29T11:40:20.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.432+0000 7f83e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:20.438 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.432+0000 7f83e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:20.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.749+0000 7f83f7ec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:20.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.749+0000 7f83f7ec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:20.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.749+0000 7f83f7ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:20.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.749+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83f0136fb0 con 0x7f83f0071c30 2024-03-29T11:40:20.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.751+0000 7f83e67fc640 1 -- 172.21.15.33:0/3118476818 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f83ec052290 con 0x7f83f0071c30 2024-03-29T11:40:20.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.751+0000 7f83e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:20.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.751+0000 7f83e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:20.753 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:20.753 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 10 librados: watch_flush enter 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 10 librados: watch_flush exit 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 >> v1:172.21.15.153:6816/2737028074 conn(0x7f83d406dcb0 legacy=0x7f83d4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 10 monclient: shutdown 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 >> v1:172.21.15.33:6790/0 conn(0x7f83f0071c30 legacy=0x7f83f00b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:20.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 shutdown_connections 2024-03-29T11:40:20.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 >> 172.21.15.33:0/3118476818 conn(0x7f83f006b660 msgr2=0x7f83f013e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:20.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 shutdown_connections 2024-03-29T11:40:20.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 1 -- 172.21.15.33:0/3118476818 wait complete. 2024-03-29T11:40:20.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:20.754+0000 7f83f7ec9640 1 librados: shutdown 2024-03-29T11:40:20.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:27.578 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:27.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.729+0000 7fcf02293640 10 monclient: get_monmap_and_config 2024-03-29T11:40:27.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.729+0000 7fcf02293640 10 monclient: build_initial_monmap 2024-03-29T11:40:27.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.729+0000 7fcf02293640 10 monclient: monmap: 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:27.731346+0000 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:27.731346+0000 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:27.733 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:27.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.730+0000 7fcf02293640 1 Processor -- start 2024-03-29T11:40:27.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.730+0000 7fcf02293640 1 -- start start 2024-03-29T11:40:27.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.730+0000 7fcf02293640 10 monclient: init 2024-03-29T11:40:27.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.731+0000 7fcf02293640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:27.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.731+0000 7fcf02293640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:27.735 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 10 monclient(hunting): picked mon.noname-c con 0x7fcefc0b0770 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:27.735 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 10 monclient(hunting): picked mon.noname-b con 0x7fcefc14a700 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:27.735 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 10 monclient(hunting): picked mon.noname-a con 0x7fcefc0ac5a0 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:27.735 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc0aba70 con 0x7fcefc0ac5a0 2024-03-29T11:40:27.735 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc0aecb0 con 0x7fcefc0b0770 2024-03-29T11:40:27.735 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc0af150 con 0x7fcefc14a700 2024-03-29T11:40:27.735 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcf02293640 10 monclient(hunting): authenticate will time out at 1183.996842s 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcefbfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcefc0b0770 0x7fcefc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:38052/0 (socket says 172.21.15.33:38052) 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.732+0000 7fcefbfff640 1 -- 172.21.15.33:0/3126740214 learned_addr learned my addr 172.21.15.33:0/3126740214 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.733+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093887210 0 0) 0x7fcefc0aecb0 con 0x7fcefc0b0770 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.733+0000 7fcefaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.733+0000 7fcefaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.733+0000 7fcefaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.733+0000 7fcefaffd640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.733+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4003520 con 0x7fcefc0b0770 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.734+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356165286 0 0) 0x7fcee4003520 con 0x7fcefc0b0770 2024-03-29T11:40:27.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.734+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 >> v1:172.21.15.33:6790/0 conn(0x7fcefc14a700 legacy=0x7fcefc0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:27.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.735+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 >> v1:172.21.15.33:6789/0 conn(0x7fcefc0ac5a0 legacy=0x7fcefc151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 1 monclient: found mon.noname-c 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0ac9d0 con 0x7fcefc0b0770 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.736+0000 7fcefaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:27.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fceec0016b0 con 0x7fcefc0b0770 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 10 monclient: dump: 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:27.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcf02293640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcf02293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fceec001940 con 0x7fcefc0b0770 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 1 -- 172.21.15.33:0/3126740214 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fceec001cd0 con 0x7fcefc0b0770 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcefaffd640 10 monclient: dump: 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:27.743 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcf02293640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.737+0000 7fcf02293640 4 set_mon_vals no callback set 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.738+0000 7fcf02293640 10 monclient: shutdown 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.738+0000 7fcf02293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.738+0000 7fcf02293640 1 -- 172.21.15.33:0/3126740214 >> v1:172.21.15.153:6789/0 conn(0x7fcefc0b0770 legacy=0x7fcefc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.738+0000 7fcf02293640 1 -- 172.21.15.33:0/3126740214 shutdown_connections 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.738+0000 7fcf02293640 1 -- 172.21.15.33:0/3126740214 >> 172.21.15.33:0/3126740214 conn(0x7fcefc06b660 msgr2=0x7fcefc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:27.744 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.738+0000 7fcf02293640 1 -- 172.21.15.33:0/3126740214 shutdown_connections 2024-03-29T11:40:27.745 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.738+0000 7fcf02293640 1 -- 172.21.15.33:0/3126740214 wait complete. 2024-03-29T11:40:27.745 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.739+0000 7fcf02293640 10 monclient: build_initial_monmap 2024-03-29T11:40:27.745 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.739+0000 7fcf02293640 10 monclient: monmap: 2024-03-29T11:40:27.745 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:27.745 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.740+0000 7fcf02293640 1 librados: starting msgr at 2024-03-29T11:40:27.746 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.740+0000 7fcf02293640 1 librados: starting objecter 2024-03-29T11:40:27.747 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.740+0000 7fcf02293640 1 Processor -- start 2024-03-29T11:40:27.747 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.742+0000 7fcf02293640 1 -- start start 2024-03-29T11:40:27.747 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.742+0000 7fcf02293640 1 librados: setting wanted keys 2024-03-29T11:40:27.747 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.742+0000 7fcf02293640 1 librados: calling monclient init 2024-03-29T11:40:27.747 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.742+0000 7fcf02293640 10 monclient: init 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 10 monclient(hunting): picked mon.noname-b con 0x7fcefc0ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 10 monclient(hunting): picked mon.noname-c con 0x7fcefc0b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 10 monclient(hunting): picked mon.noname-a con 0x7fcefc14a700 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc1239d0 con 0x7fcefc14a700 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc125cb0 con 0x7fcefc0ac5a0 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc145930 con 0x7fcefc0b0770 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:27.749 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.744+0000 7fcf02293640 10 monclient(hunting): authenticate will time out at 1184.009368s 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.745+0000 7fcefbfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcefc0ac5a0 0x7fcefc144cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:39138/0 (socket says 172.21.15.33:39138) 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.745+0000 7fcefbfff640 1 -- 172.21.15.33:0/124286015 learned_addr learned my addr 172.21.15.33:0/124286015 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.746+0000 7fcef8ff9640 10 client.?.objecter ms_handle_connect 0x7fcefc0ac5a0 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.746+0000 7fcef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.748+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934148125 0 0) 0x7fcefc125cb0 con 0x7fcefc0ac5a0 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.748+0000 7fcef8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:27.750 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.748+0000 7fcef8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.748+0000 7fcef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.748+0000 7fcef8ff9640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:40:27.751 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.748+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee8004eb0 con 0x7fcefc0ac5a0 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.748+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210847448 0 0) 0x7fcee8004eb0 con 0x7fcefc0ac5a0 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcefc125cb0 con 0x7fcefc0ac5a0 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fceec001570 con 0x7fcefc0ac5a0 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:27.752 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 client.?.objecter ms_handle_connect 0x7fcefc0b0770 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844575499 0 0) 0x7fcefc145930 con 0x7fcefc0b0770 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:27.753 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:40:27.754 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.749+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee8004eb0 con 0x7fcefc0b0770 2024-03-29T11:40:27.755 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3939029663 0 0) 0x7fcee8004eb0 con 0x7fcefc0b0770 2024-03-29T11:40:27.757 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee8005990 con 0x7fcefc0b0770 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fcef00016b0 con 0x7fcefc0b0770 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.758 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 699141891 0 0) 0x7fcee8005990 con 0x7fcefc0b0770 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 >> v1:172.21.15.153:6789/0 conn(0x7fcefc0ac5a0 legacy=0x7fcefc144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 >> v1:172.21.15.33:6789/0 conn(0x7fcefc14a700 legacy=0x7fcefc1e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 monclient: found mon.c 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc1453c0 con 0x7fcefc0b0770 2024-03-29T11:40:27.759 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcef8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.750+0000 7fcf02293640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 10 monclient: _renew_subs 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc145630 con 0x7fcefc0b0770 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 10 monclient: _renew_subs 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc126b00 con 0x7fcefc0b0770 2024-03-29T11:40:27.760 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcf02293640 1 librados: init done 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef0001c10 con 0x7fcefc0b0770 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcef8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fcef00016b0 con 0x7fcefc0b0770 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcef8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcef8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.751+0000 7fcef8ff9640 10 monclient: dump: 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:27.761 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.752+0000 7fcefa7fc640 4 set_mon_vals no callback set 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.754+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fcef0019460 con 0x7fcefc0b0770 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcefbfff640 10 monclient: get_auth_request con 0x7fcee806e290 auth_method 0 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fcef00530e0 con 0x7fcefc0b0770 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcef8ff9640 10 client.4475.objecter ms_dispatch 0x7fcefc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:27.763 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcef8ff9640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcef8ff9640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcef8ff9640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcf02293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcf02293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcf02293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.755+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc005080 con 0x7fcefc0b0770 2024-03-29T11:40:27.764 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.756+0000 7fcef8ff9640 10 client.4475.objecter ms_handle_connect 0x7fcee806e290 2024-03-29T11:40:27.767 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.765+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcef004f380 con 0x7fcefc0b0770 2024-03-29T11:40:27.768 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.765+0000 7fcef8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:27.768 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:27.765+0000 7fcef8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:28.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.113+0000 7fcf02293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:28.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.113+0000 7fcf02293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:28.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.113+0000 7fcf02293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:28.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.113+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcedc005330 con 0x7fcefc0b0770 2024-03-29T11:40:28.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.114+0000 7fcef8ff9640 1 -- 172.21.15.33:0/124286015 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fcef004f520 con 0x7fcefc0b0770 2024-03-29T11:40:28.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.114+0000 7fcef8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:28.116 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.114+0000 7fcef8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:28.116 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:28.117 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:28.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 10 librados: watch_flush enter 2024-03-29T11:40:28.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 10 librados: watch_flush exit 2024-03-29T11:40:28.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:40:28.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:40:28.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 >> v1:172.21.15.153:6816/2737028074 conn(0x7fcee806e290 legacy=0x7fcee8070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:28.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 10 monclient: shutdown 2024-03-29T11:40:28.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:28.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.117+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 >> v1:172.21.15.33:6790/0 conn(0x7fcefc0b0770 legacy=0x7fcefc1deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:28.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.118+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 shutdown_connections 2024-03-29T11:40:28.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.118+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 >> 172.21.15.33:0/124286015 conn(0x7fcefc06b660 msgr2=0x7fcefc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:28.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.119+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 shutdown_connections 2024-03-29T11:40:28.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.119+0000 7fcf02293640 1 -- 172.21.15.33:0/124286015 wait complete. 2024-03-29T11:40:28.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:28.120+0000 7fcf02293640 1 librados: shutdown 2024-03-29T11:40:28.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:34.947 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.097+0000 7fbf03fff640 10 monclient: get_monmap_and_config 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.097+0000 7fbf03fff640 10 monclient: build_initial_monmap 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.097+0000 7fbf03fff640 10 monclient: monmap: 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:35.098230+0000 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:35.098230+0000 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:35.099 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:35.100 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:35.100 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:35.100 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:35.100 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.098+0000 7fbf03fff640 1 Processor -- start 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.098+0000 7fbf03fff640 1 -- start start 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.098+0000 7fbf03fff640 10 monclient: init 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 10 monclient(hunting): picked mon.noname-a con 0x7fbefc038c60 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 10 monclient(hunting): picked mon.noname-b con 0x7fbefc0324b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 10 monclient(hunting): picked mon.noname-c con 0x7fbefc0d11e0 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc0caf80 con 0x7fbefc038c60 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc039850 con 0x7fbefc0d11e0 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc0c78d0 con 0x7fbefc0324b0 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:35.102 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.099+0000 7fbf03fff640 10 monclient(hunting): authenticate will time out at 1191.363300s 2024-03-29T11:40:35.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.103+0000 7fbf027fc640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fbefc0324b0 0x7fbefc0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:52595/0 (socket says 172.21.15.33:52595) 2024-03-29T11:40:35.104 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.103+0000 7fbf027fc640 1 -- 172.21.15.33:0/3874131808 learned_addr learned my addr 172.21.15.33:0/3874131808 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:35.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.104+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105835515 0 0) 0x7fbefc0caf80 con 0x7fbefc038c60 2024-03-29T11:40:35.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:35.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:35.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:35.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:40:35.106 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec003520 con 0x7fbefc038c60 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705550614 0 0) 0x7fbeec003520 con 0x7fbefc038c60 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 >> v1:172.21.15.33:6790/0 conn(0x7fbefc0324b0 legacy=0x7fbefc0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 >> v1:172.21.15.153:6789/0 conn(0x7fbefc0d11e0 legacy=0x7fbefc0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 1 monclient: found mon.noname-a 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:40:35.107 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc0c7a60 con 0x7fbefc038c60 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbef8001690 con 0x7fbefc038c60 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.105+0000 7fbf01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf01ffb640 10 monclient: dump: 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:35.108 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef80018b0 con 0x7fbefc038c60 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf01ffb640 1 -- 172.21.15.33:0/3874131808 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbef8001c40 con 0x7fbefc038c60 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf01ffb640 10 monclient: dump: 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:35.109 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:35.110 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:35.110 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:35.110 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:35.111 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf03fff640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:40:35.111 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf03fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:35.111 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.106+0000 7fbf03fff640 4 set_mon_vals no callback set 2024-03-29T11:40:35.111 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.107+0000 7fbf03fff640 10 monclient: shutdown 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.107+0000 7fbf03fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.107+0000 7fbf03fff640 1 -- 172.21.15.33:0/3874131808 >> v1:172.21.15.33:6789/0 conn(0x7fbefc038c60 legacy=0x7fbefc0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.107+0000 7fbf03fff640 1 -- 172.21.15.33:0/3874131808 shutdown_connections 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.107+0000 7fbf03fff640 1 -- 172.21.15.33:0/3874131808 >> 172.21.15.33:0/3874131808 conn(0x7fbefc005c20 msgr2=0x7fbefc006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.107+0000 7fbf03fff640 1 -- 172.21.15.33:0/3874131808 shutdown_connections 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.107+0000 7fbf03fff640 1 -- 172.21.15.33:0/3874131808 wait complete. 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.109+0000 7fbf03fff640 10 monclient: build_initial_monmap 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.109+0000 7fbf03fff640 10 monclient: monmap: 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:35.112 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.109+0000 7fbf03fff640 1 librados: starting msgr at 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.109+0000 7fbf03fff640 1 librados: starting objecter 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.109+0000 7fbf03fff640 1 Processor -- start 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.110+0000 7fbf03fff640 1 -- start start 2024-03-29T11:40:35.113 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.110+0000 7fbf03fff640 1 librados: setting wanted keys 2024-03-29T11:40:35.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.110+0000 7fbf03fff640 1 librados: calling monclient init 2024-03-29T11:40:35.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.110+0000 7fbf03fff640 10 monclient: init 2024-03-29T11:40:35.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.111+0000 7fbf03fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:35.114 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.111+0000 7fbf03fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:35.115 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.111+0000 7fbf03fff640 10 monclient(hunting): picked mon.noname-a con 0x7fbefc0324b0 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:35.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.113+0000 7fbf02ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fbefc0324b0 0x7fbefc0346b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:43567/0 (socket says 172.21.15.33:43567) 2024-03-29T11:40:35.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.116+0000 7fbf027fc640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fbefc0d11e0 0x7fbefc0bc010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:52671/0 (socket says 172.21.15.33:52671) 2024-03-29T11:40:35.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.116+0000 7fbf03fff640 10 monclient(hunting): picked mon.noname-c con 0x7fbefc0d11e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:35.118 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.116+0000 7fbf02ffd640 1 -- 172.21.15.33:0/4145844761 learned_addr learned my addr 172.21.15.33:0/4145844761 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.117+0000 7fbf03fff640 10 monclient(hunting): picked mon.noname-b con 0x7fbefc0ca850 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.117+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc036090 con 0x7fbefc0324b0 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.117+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc0bce30 con 0x7fbefc0ca850 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.117+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc036300 con 0x7fbefc0d11e0 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.117+0000 7fbf03fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.117+0000 7fbf03fff640 10 monclient(hunting): authenticate will time out at 1191.381331s 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.117+0000 7fbeeb7fe640 10 client.?.objecter ms_handle_connect 0x7fbefc0d11e0 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.118+0000 7fbeeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.118+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543608503 0 0) 0x7fbefc036300 con 0x7fbefc0d11e0 2024-03-29T11:40:35.120 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.118+0000 7fbeeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.118+0000 7fbeeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.118+0000 7fbeeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.118+0000 7fbeeb7fe640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.118+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef0004eb0 con 0x7fbefc0d11e0 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.120+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375867995 0 0) 0x7fbef0004eb0 con 0x7fbefc0d11e0 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.120+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbefc036300 con 0x7fbefc0d11e0 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.120+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbef4001930 con 0x7fbefc0d11e0 2024-03-29T11:40:35.121 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.120+0000 7fbeeb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.120+0000 7fbeeb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.120+0000 7fbeeb7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:35.122 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.120+0000 7fbeeb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:35.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 client.?.objecter ms_handle_connect 0x7fbefc0324b0 2024-03-29T11:40:35.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:35.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4029233750 0 0) 0x7fbefc036300 con 0x7fbefc0d11e0 2024-03-29T11:40:35.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 >> v1:172.21.15.153:6789/0 conn(0x7fbefc0ca850 legacy=0x7fbefc0bc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.123 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 >> v1:172.21.15.33:6789/0 conn(0x7fbefc0324b0 legacy=0x7fbefc0346b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 1 monclient: found mon.c 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc034dc0 con 0x7fbefc0d11e0 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 client.?.objecter ms_handle_connect 0x7fbefc0ca850 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 10 monclient: _renew_subs 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbefc035020 con 0x7fbefc0d11e0 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:35.124 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 10 monclient: _renew_subs 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbefc0c0c70 con 0x7fbefc0d11e0 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbf03fff640 1 librados: init done 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.121+0000 7fbeeb7fe640 10 client.4478.objecter resend_mon_ops 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.123+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef4001a90 con 0x7fbefc0d11e0 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.123+0000 7fbeeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.123+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fbef4004ac0 con 0x7fbefc0d11e0 2024-03-29T11:40:35.125 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.123+0000 7fbeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.123+0000 7fbeeb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.123+0000 7fbeeb7fe640 10 monclient: dump: 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.124+0000 7fbf03fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.124+0000 7fbf03fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:35.126 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.124+0000 7fbf03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:35.127 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.124+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbefc007b90 con 0x7fbefc0d11e0 2024-03-29T11:40:35.127 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.125+0000 7fbf017fa640 4 set_mon_vals no callback set 2024-03-29T11:40:35.127 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.126+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fbef4004c20 con 0x7fbefc0d11e0 2024-03-29T11:40:35.129 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.127+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fbef4084ad0 con 0x7fbefc0d11e0 2024-03-29T11:40:35.129 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.127+0000 7fbeeb7fe640 10 client.4478.objecter ms_dispatch 0x7fbf04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:35.129 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.127+0000 7fbeeb7fe640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:35.129 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.127+0000 7fbeeb7fe640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:35.129 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.127+0000 7fbeeb7fe640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T11:40:35.134 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.130+0000 7fbf02ffd640 10 monclient: get_auth_request con 0x7fbef006dd40 auth_method 0 2024-03-29T11:40:35.135 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.131+0000 7fbeeb7fe640 10 client.4478.objecter ms_handle_connect 0x7fbef006dd40 2024-03-29T11:40:35.139 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.137+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbef4086e60 con 0x7fbefc0d11e0 2024-03-29T11:40:35.139 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.137+0000 7fbeeb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:35.139 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.137+0000 7fbeeb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:35.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.460+0000 7fbf03fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:35.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.460+0000 7fbf03fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:35.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.460+0000 7fbf03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:35.462 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.460+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbefc0343d0 con 0x7fbefc0d11e0 2024-03-29T11:40:35.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.461+0000 7fbeeb7fe640 1 -- 172.21.15.33:0/4145844761 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fbefc0343d0 con 0x7fbefc0d11e0 2024-03-29T11:40:35.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.461+0000 7fbeeb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:35.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.461+0000 7fbeeb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:35.463 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:35.463 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:35.466 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.464+0000 7fbf03fff640 10 librados: watch_flush enter 2024-03-29T11:40:35.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.467+0000 7fbf03fff640 10 librados: watch_flush exit 2024-03-29T11:40:35.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.467+0000 7fbf03fff640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T11:40:35.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.467+0000 7fbf03fff640 10 client.4478.objecter successfully canceled tick 2024-03-29T11:40:35.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.468+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 >> v1:172.21.15.153:6816/2737028074 conn(0x7fbef006dd40 legacy=0x7fbef0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.468+0000 7fbf03fff640 10 monclient: shutdown 2024-03-29T11:40:35.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.468+0000 7fbf03fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:35.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.468+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 >> v1:172.21.15.33:6790/0 conn(0x7fbefc0d11e0 legacy=0x7fbefc0bc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:35.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.469+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 shutdown_connections 2024-03-29T11:40:35.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.469+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 >> 172.21.15.33:0/4145844761 conn(0x7fbefc005c20 msgr2=0x7fbefc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:35.473 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.472+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 shutdown_connections 2024-03-29T11:40:35.473 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.472+0000 7fbf03fff640 1 -- 172.21.15.33:0/4145844761 wait complete. 2024-03-29T11:40:35.474 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:35.472+0000 7fbf03fff640 1 librados: shutdown 2024-03-29T11:40:35.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:42.298 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.453+0000 7f5947a74640 10 monclient: get_monmap_and_config 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.453+0000 7f5947a74640 10 monclient: build_initial_monmap 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.453+0000 7f5947a74640 10 monclient: monmap: 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:42.453945+0000 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:42.453945+0000 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:42.455 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:42.456 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:42.456 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.459 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.458+0000 7f5947a74640 1 Processor -- start 2024-03-29T11:40:42.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.458+0000 7f5947a74640 1 -- start start 2024-03-29T11:40:42.460 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.458+0000 7f5947a74640 10 monclient: init 2024-03-29T11:40:42.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:42.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:42.463 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 10 monclient(hunting): picked mon.noname-a con 0x7f59400b0770 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:42.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 10 monclient(hunting): picked mon.noname-c con 0x7f594014a700 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:42.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 10 monclient(hunting): picked mon.noname-b con 0x7f59400ac5a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:42.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59400aba70 con 0x7f59400b0770 2024-03-29T11:40:42.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59400aecb0 con 0x7f594014a700 2024-03-29T11:40:42.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59400af150 con 0x7f59400ac5a0 2024-03-29T11:40:42.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:42.464 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.459+0000 7f5947a74640 10 monclient(hunting): authenticate will time out at 1198.723400s 2024-03-29T11:40:42.466 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.464+0000 7f594500f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f594014a700 0x7f59400abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:58703/0 (socket says 172.21.15.33:58703) 2024-03-29T11:40:42.466 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.464+0000 7f594500f640 1 -- 172.21.15.33:0/856423021 learned_addr learned my addr 172.21.15.33:0/856423021 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:42.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456330696 0 0) 0x7f59400aecb0 con 0x7f594014a700 2024-03-29T11:40:42.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 10 monclient(hunting): _negotiate 2024-03-29T11:40:42.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:42.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:42.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:40:42.467 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f592c003520 con 0x7f594014a700 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 469615907 0 0) 0x7f592c003520 con 0x7f594014a700 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 >> v1:172.21.15.33:6790/0 conn(0x7f59400ac5a0 legacy=0x7f5940151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.465+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 >> v1:172.21.15.33:6789/0 conn(0x7f59400b0770 legacy=0x7f5940149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: _finish_hunting 0 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 1 monclient: found mon.noname-c 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400ac9d0 con 0x7f594014a700 2024-03-29T11:40:42.468 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: _finish_auth 0 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: _check_auth_tickets 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f593c0016b0 con 0x7f594014a700 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: dump: 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:42.469 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f593c001880 con 0x7f594014a700 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 1 -- 172.21.15.33:0/856423021 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f593c001cd0 con 0x7f594014a700 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.466+0000 7f594480e640 10 monclient: dump: 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:42.470 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:42.471 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.471 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.468+0000 7f5947a74640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:40:42.471 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.468+0000 7f5947a74640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.468+0000 7f5947a74640 4 set_mon_vals no callback set 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.469+0000 7f5947a74640 10 monclient: shutdown 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.469+0000 7f5947a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.469+0000 7f5947a74640 1 -- 172.21.15.33:0/856423021 >> v1:172.21.15.153:6789/0 conn(0x7f594014a700 legacy=0x7f59400abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.469+0000 7f5947a74640 1 -- 172.21.15.33:0/856423021 shutdown_connections 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.469+0000 7f5947a74640 1 -- 172.21.15.33:0/856423021 >> 172.21.15.33:0/856423021 conn(0x7f594006b660 msgr2=0x7f594006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.469+0000 7f5947a74640 1 -- 172.21.15.33:0/856423021 shutdown_connections 2024-03-29T11:40:42.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.470+0000 7f5947a74640 1 -- 172.21.15.33:0/856423021 wait complete. 2024-03-29T11:40:42.473 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.472+0000 7f5947a74640 10 monclient: build_initial_monmap 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.472+0000 7f5947a74640 10 monclient: monmap: 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.475 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.473+0000 7f5947a74640 1 librados: starting msgr at 2024-03-29T11:40:42.476 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.473+0000 7f5947a74640 1 librados: starting objecter 2024-03-29T11:40:42.476 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.473+0000 7f5947a74640 1 Processor -- start 2024-03-29T11:40:42.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.473+0000 7f5947a74640 1 -- start start 2024-03-29T11:40:42.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.473+0000 7f5947a74640 1 librados: setting wanted keys 2024-03-29T11:40:42.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.473+0000 7f5947a74640 1 librados: calling monclient init 2024-03-29T11:40:42.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.473+0000 7f5947a74640 10 monclient: init 2024-03-29T11:40:42.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 10 monclient(hunting): picked mon.noname-a con 0x7f59400ac5a0 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 10 monclient(hunting): picked mon.noname-c con 0x7f59400b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 10 monclient(hunting): picked mon.noname-b con 0x7f594014a700 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594012aba0 con 0x7f59400ac5a0 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59401287b0 con 0x7f594014a700 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5940129930 con 0x7f59400b0770 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:42.478 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.474+0000 7f5947a74640 10 monclient(hunting): authenticate will time out at 1198.738213s 2024-03-29T11:40:42.481 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.478+0000 7f5946011640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f594014a700 0x7f59401db6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:58761/0 (socket says 172.21.15.33:58761) 2024-03-29T11:40:42.481 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.478+0000 7f5946011640 1 -- 172.21.15.33:0/2447002207 learned_addr learned my addr 172.21.15.33:0/2447002207 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:42.481 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.478+0000 7f59367fc640 10 client.?.objecter ms_handle_connect 0x7f594014a700 2024-03-29T11:40:42.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.478+0000 7f59367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:42.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.479+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810664994 0 0) 0x7f59401287b0 con 0x7f594014a700 2024-03-29T11:40:42.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.479+0000 7f59367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:42.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.479+0000 7f59367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:42.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.479+0000 7f59367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:42.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.479+0000 7f59367fc640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:40:42.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.479+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5924004eb0 con 0x7f594014a700 2024-03-29T11:40:42.484 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 client.?.objecter ms_handle_connect 0x7f59400b0770 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 client.?.objecter ms_handle_connect 0x7f59400ac5a0 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845979180 0 0) 0x7f5940129930 con 0x7f59400b0770 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59401287b0 con 0x7f59400b0770 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.481+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 658584267 0 0) 0x7f5924004eb0 con 0x7f594014a700 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5940129930 con 0x7f594014a700 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5938001c60 con 0x7f594014a700 2024-03-29T11:40:42.485 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): dump: 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:42.486 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45035587 0 0) 0x7f59401287b0 con 0x7f59400b0770 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5924004eb0 con 0x7f59400b0770 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f593c001b90 con 0x7f59400b0770 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): dump: 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:42.487 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629672845 0 0) 0x7f594012aba0 con 0x7f59400ac5a0 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59401287b0 con 0x7f59400ac5a0 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3771534539 0 0) 0x7f59401287b0 con 0x7f59400ac5a0 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.482+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f594012aba0 con 0x7f59400ac5a0 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5930001700 con 0x7f59400ac5a0 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 10 monclient(hunting): dump: 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:42.488 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 532877522 0 0) 0x7f5924004eb0 con 0x7f59400b0770 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 >> v1:172.21.15.153:6789/0 conn(0x7f594014a700 legacy=0x7f59401db6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 >> v1:172.21.15.33:6789/0 conn(0x7f59400ac5a0 legacy=0x7f5940129220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 1 monclient: found mon.c 2024-03-29T11:40:42.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:42.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.483+0000 7f59367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:42.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59401485c0 con 0x7f59400b0770 2024-03-29T11:40:42.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f59367fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:42.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f59367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:42.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f59367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 10 monclient: _renew_subs 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5940148050 con 0x7f59400b0770 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 10 monclient: _renew_subs 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59401dbdc0 con 0x7f59400b0770 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.484+0000 7f5947a74640 1 librados: init done 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.486+0000 7f5947a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.486+0000 7f5947a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.486+0000 7f5947a74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.486+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59401db3d0 con 0x7f59400b0770 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.487+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f593c001d70 con 0x7f59400b0770 2024-03-29T11:40:42.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.487+0000 7f59367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.487+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f593c001250 con 0x7f59400b0770 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.487+0000 7f59367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.487+0000 7f59367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.487+0000 7f59367fc640 10 monclient: dump: 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:42.493 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.487+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f593c01aae0 con 0x7f59400b0770 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.488+0000 7f5937fff640 4 set_mon_vals no callback set 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.488+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f593c084a60 con 0x7f59400b0770 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.488+0000 7f59367fc640 10 client.4481.objecter ms_dispatch 0x7f5940000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.488+0000 7f59367fc640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.488+0000 7f59367fc640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.488+0000 7f59367fc640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.488+0000 7f5945810640 10 monclient: get_auth_request con 0x7f592406e250 auth_method 0 2024-03-29T11:40:42.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.489+0000 7f59367fc640 10 client.4481.objecter ms_handle_connect 0x7f592406e250 2024-03-29T11:40:42.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.495+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59401db3d0 con 0x7f59400b0770 2024-03-29T11:40:42.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.495+0000 7f59367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:42.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.495+0000 7f59367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:42.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.828+0000 7f5947a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.828+0000 7f5947a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.828+0000 7f5947a74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.828+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59401db3d0 con 0x7f59400b0770 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.829+0000 7f59367fc640 1 -- 172.21.15.33:0/2447002207 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f59401db3d0 con 0x7f59400b0770 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.829+0000 7f59367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.829+0000 7f59367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:42.832 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:42.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.833+0000 7f5947a74640 10 librados: watch_flush enter 2024-03-29T11:40:42.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.834+0000 7f5947a74640 10 librados: watch_flush exit 2024-03-29T11:40:42.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.834+0000 7f5947a74640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:40:42.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.834+0000 7f5947a74640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:40:42.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.835+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 >> v1:172.21.15.153:6816/2737028074 conn(0x7f592406e250 legacy=0x7f5924070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.835+0000 7f5947a74640 10 monclient: shutdown 2024-03-29T11:40:42.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.835+0000 7f5947a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:42.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.835+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 >> v1:172.21.15.33:6790/0 conn(0x7f59400b0770 legacy=0x7f5940147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:42.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.836+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 shutdown_connections 2024-03-29T11:40:42.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.836+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 >> 172.21.15.33:0/2447002207 conn(0x7f594006b660 msgr2=0x7f594006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:42.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.837+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 shutdown_connections 2024-03-29T11:40:42.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.837+0000 7f5947a74640 1 -- 172.21.15.33:0/2447002207 wait complete. 2024-03-29T11:40:42.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:42.837+0000 7f5947a74640 1 librados: shutdown 2024-03-29T11:40:42.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:49.664 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:49.814 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.813+0000 7f328d9e1640 10 monclient: get_monmap_and_config 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.813+0000 7f328d9e1640 10 monclient: build_initial_monmap 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.813+0000 7f328d9e1640 10 monclient: monmap: 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:49.813758+0000 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:49.813758+0000 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:49.815 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.815+0000 7f328d9e1640 1 Processor -- start 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.815+0000 7f328d9e1640 1 -- start start 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.815+0000 7f328d9e1640 10 monclient: init 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 10 monclient(hunting): picked mon.noname-a con 0x7f32800389e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 10 monclient(hunting): picked mon.noname-c con 0x7f3280038dc0 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 10 monclient(hunting): picked mon.noname-b con 0x7f3280030660 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32800c54c0 con 0x7f32800389e0 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3280036430 con 0x7f3280038dc0 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3280039840 con 0x7f3280030660 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:49.818 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.816+0000 7f328d9e1640 10 monclient(hunting): authenticate will time out at 1206.080088s 2024-03-29T11:40:49.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.821+0000 7f328c9df640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f32800389e0 0x7f3280033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:42831/0 (socket says 172.21.15.33:42831) 2024-03-29T11:40:49.823 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.821+0000 7f328c9df640 1 -- 172.21.15.33:0/274319961 learned_addr learned my addr 172.21.15.33:0/274319961 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:49.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.822+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508349965 0 0) 0x7f32800c54c0 con 0x7f32800389e0 2024-03-29T11:40:49.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.822+0000 7f32877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.822+0000 7f32877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.822+0000 7f32877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.822+0000 7f32877fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:40:49.824 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.822+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3270003520 con 0x7f32800389e0 2024-03-29T11:40:49.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681686976 0 0) 0x7f3280039840 con 0x7f3280030660 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32800c54c0 con 0x7f3280030660 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125188343 0 0) 0x7f3270003520 con 0x7f32800389e0 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 >> v1:172.21.15.33:6790/0 conn(0x7f3280030660 legacy=0x7f32800d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 >> v1:172.21.15.153:6789/0 conn(0x7f3280038dc0 legacy=0x7f32800cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 monclient: found mon.noname-a 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3280035690 con 0x7f32800389e0 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:49.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3278001690 con 0x7f32800389e0 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f328d9e1640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.826+0000 7f328d9e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 10 monclient: dump: 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:49.831 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f328d9e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3278001860 con 0x7f32800389e0 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 1 -- 172.21.15.33:0/274319961 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3278001cb0 con 0x7f32800389e0 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f32877fe640 10 monclient: dump: 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:49.832 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:49.833 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:49.833 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:49.833 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:49.833 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f328d9e1640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:49.833 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.827+0000 7f328d9e1640 4 set_mon_vals no callback set 2024-03-29T11:40:49.834 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.828+0000 7f328d9e1640 10 monclient: shutdown 2024-03-29T11:40:49.834 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.828+0000 7f328d9e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:49.834 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.828+0000 7f328d9e1640 1 -- 172.21.15.33:0/274319961 >> v1:172.21.15.33:6789/0 conn(0x7f32800389e0 legacy=0x7f3280033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.834 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.828+0000 7f328d9e1640 1 -- 172.21.15.33:0/274319961 shutdown_connections 2024-03-29T11:40:49.834 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.828+0000 7f328d9e1640 1 -- 172.21.15.33:0/274319961 >> 172.21.15.33:0/274319961 conn(0x7f3280005c00 msgr2=0x7f3280006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:49.834 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.833+0000 7f328d9e1640 1 -- 172.21.15.33:0/274319961 shutdown_connections 2024-03-29T11:40:49.835 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.833+0000 7f328d9e1640 1 -- 172.21.15.33:0/274319961 wait complete. 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.834+0000 7f328d9e1640 10 monclient: build_initial_monmap 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.834+0000 7f328d9e1640 10 monclient: monmap: 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.834+0000 7f328d9e1640 1 librados: starting msgr at 2024-03-29T11:40:49.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.834+0000 7f328d9e1640 1 librados: starting objecter 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.835+0000 7f328d9e1640 1 Processor -- start 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.835+0000 7f328d9e1640 1 -- start start 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.835+0000 7f328d9e1640 1 librados: setting wanted keys 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.835+0000 7f328d9e1640 1 librados: calling monclient init 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.835+0000 7f328d9e1640 10 monclient: init 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 10 monclient(hunting): picked mon.noname-b con 0x7f3280030660 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 10 monclient(hunting): picked mon.noname-c con 0x7f32800c7750 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 10 monclient(hunting): picked mon.noname-a con 0x7f328015d470 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32800c6a90 con 0x7f328015d470 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32800c6d70 con 0x7f3280030660 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32800c7050 con 0x7f32800c7750 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:49.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.836+0000 7f328d9e1640 10 monclient(hunting): authenticate will time out at 1206.100267s 2024-03-29T11:40:49.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f328c9df640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3280030660 0x7f328015cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:59109/0 (socket says 172.21.15.33:59109) 2024-03-29T11:40:49.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f328c9df640 1 -- 172.21.15.33:0/1272127612 learned_addr learned my addr 172.21.15.33:0/1272127612 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:49.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 10 client.?.objecter ms_handle_connect 0x7f3280030660 2024-03-29T11:40:49.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:49.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819452869 0 0) 0x7f32800c6d70 con 0x7f3280030660 2024-03-29T11:40:49.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:40:49.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.841+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3274004eb0 con 0x7f3280030660 2024-03-29T11:40:49.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.842+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1246867735 0 0) 0x7f3274004eb0 con 0x7f3280030660 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32800c6d70 con 0x7f3280030660 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 client.?.objecter ms_handle_connect 0x7f32800c7750 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3278001b70 con 0x7f3280030660 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): dump: 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:49.846 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657413441 0 0) 0x7f32800c7050 con 0x7f32800c7750 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:40:49.847 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.844+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3274004eb0 con 0x7f32800c7750 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.845+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3175348625 0 0) 0x7f32800c6d70 con 0x7f3280030660 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.845+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 >> v1:172.21.15.33:6790/0 conn(0x7f32800c7750 legacy=0x7f32800c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.845+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 >> v1:172.21.15.33:6789/0 conn(0x7f328015d470 legacy=0x7f32801680c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 1 monclient: found mon.b 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328015c840 con 0x7f3280030660 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f32857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:49.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:40:49.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 10 monclient: _renew_subs 2024-03-29T11:40:49.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:49.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32800c5b00 con 0x7f3280030660 2024-03-29T11:40:49.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 10 monclient: _renew_subs 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32800c84c0 con 0x7f3280030660 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.846+0000 7f328d9e1640 1 librados: init done 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.849+0000 7f328d9e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.849+0000 7f328d9e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.849+0000 7f328d9e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.849+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3280007b90 con 0x7f3280030660 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.849+0000 7f32857fa640 10 client.4459.objecter ms_handle_connect 0x7f328015d470 2024-03-29T11:40:49.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.849+0000 7f32857fa640 10 client.4459.objecter resend_mon_ops 2024-03-29T11:40:49.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.853+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3278001d10 con 0x7f3280030660 2024-03-29T11:40:49.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.853+0000 7f32857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.853+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f32780010e0 con 0x7f3280030660 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.853+0000 7f32857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.853+0000 7f32857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.853+0000 7f32857fa640 10 monclient: dump: 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:49.855 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:49.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.857+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f327801ac80 con 0x7f3280030660 2024-03-29T11:40:49.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.853+0000 7f3286ffd640 4 set_mon_vals no callback set 2024-03-29T11:40:49.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.859+0000 7f3287fff640 10 monclient: get_auth_request con 0x7f327406dc60 auth_method 0 2024-03-29T11:40:49.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.859+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f3278084af0 con 0x7f3280030660 2024-03-29T11:40:49.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.859+0000 7f32857fa640 10 client.4459.objecter ms_dispatch 0x7f3288000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:49.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.859+0000 7f32857fa640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:49.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.859+0000 7f32857fa640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:49.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.859+0000 7f32857fa640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:40:49.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.860+0000 7f32857fa640 10 client.4459.objecter ms_handle_connect 0x7f327406dc60 2024-03-29T11:40:49.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.862+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32780503d0 con 0x7f3280030660 2024-03-29T11:40:49.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.862+0000 7f32857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:49.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:49.862+0000 7f32857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:50.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.189+0000 7f328d9e1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:50.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.189+0000 7f328d9e1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:50.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.189+0000 7f328d9e1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:40:50.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.189+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3280005120 con 0x7f3280030660 2024-03-29T11:40:50.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.192+0000 7f32857fa640 1 -- 172.21.15.33:0/1272127612 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f3278052300 con 0x7f3280030660 2024-03-29T11:40:50.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.192+0000 7f32857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:50.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.192+0000 7f32857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:50.194 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:50.194 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:50.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.196+0000 7f328d9e1640 10 librados: watch_flush enter 2024-03-29T11:40:50.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.199+0000 7f328d9e1640 10 librados: watch_flush exit 2024-03-29T11:40:50.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.199+0000 7f328d9e1640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:40:50.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.199+0000 7f328d9e1640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:40:50.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.199+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 >> v1:172.21.15.153:6816/2737028074 conn(0x7f327406dc60 legacy=0x7f3274070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:50.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.199+0000 7f328d9e1640 10 monclient: shutdown 2024-03-29T11:40:50.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.199+0000 7f328d9e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:50.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.199+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 >> v1:172.21.15.153:6789/0 conn(0x7f3280030660 legacy=0x7f328015cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:50.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.202+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 shutdown_connections 2024-03-29T11:40:50.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.202+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 >> 172.21.15.33:0/1272127612 conn(0x7f3280005c00 msgr2=0x7f32800c4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:50.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.202+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 shutdown_connections 2024-03-29T11:40:50.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.202+0000 7f328d9e1640 1 -- 172.21.15.33:0/1272127612 wait complete. 2024-03-29T11:40:50.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:50.202+0000 7f328d9e1640 1 librados: shutdown 2024-03-29T11:40:50.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:57.028 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:57.175 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.174+0000 7faa459d1640 10 monclient: get_monmap_and_config 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.174+0000 7faa459d1640 10 monclient: build_initial_monmap 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.174+0000 7faa459d1640 10 monclient: monmap: 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:40:57.174758+0000 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:40:57.174758+0000 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:40:57.176 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:57.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.175+0000 7faa459d1640 1 Processor -- start 2024-03-29T11:40:57.177 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.175+0000 7faa459d1640 1 -- start start 2024-03-29T11:40:57.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.175+0000 7faa459d1640 10 monclient: init 2024-03-29T11:40:57.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:57.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:57.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 10 monclient(hunting): picked mon.noname-a con 0x7faa4014a400 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:57.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 10 monclient(hunting): picked mon.noname-b con 0x7faa400aa500 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:57.178 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 10 monclient(hunting): picked mon.noname-c con 0x7faa40149800 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:57.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400aca50 con 0x7faa4014a400 2024-03-29T11:40:57.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa40149cb0 con 0x7faa40149800 2024-03-29T11:40:57.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400acc80 con 0x7faa400aa500 2024-03-29T11:40:57.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:57.179 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.176+0000 7faa459d1640 10 monclient(hunting): authenticate will time out at 1213.440057s 2024-03-29T11:40:57.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.180+0000 7faa449cf640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7faa4014a400 0x7faa400aac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:48903/0 (socket says 172.21.15.33:48903) 2024-03-29T11:40:57.182 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.181+0000 7faa449cf640 1 -- 172.21.15.33:0/4013712057 learned_addr learned my addr 172.21.15.33:0/4013712057 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:57.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.183+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262660859 0 0) 0x7faa40149cb0 con 0x7faa40149800 2024-03-29T11:40:57.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.183+0000 7faa3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.183+0000 7faa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.183+0000 7faa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.183+0000 7faa3f7fe640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:40:57.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.183+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa30003520 con 0x7faa40149800 2024-03-29T11:40:57.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.186+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448039727 0 0) 0x7faa30003520 con 0x7faa40149800 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.186+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 >> v1:172.21.15.33:6790/0 conn(0x7faa400aa500 legacy=0x7faa400ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.186+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 >> v1:172.21.15.33:6789/0 conn(0x7faa4014a400 legacy=0x7faa400aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.186+0000 7faa3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400ac350 con 0x7faa40149800 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa459d1640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa459d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:57.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7faa38001a10 con 0x7faa40149800 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 10 monclient: dump: 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa459d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:57.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa38001b70 con 0x7faa40149800 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa459d1640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.187+0000 7faa459d1640 4 set_mon_vals no callback set 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa3f7fe640 1 -- 172.21.15.33:0/4013712057 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7faa38001fc0 con 0x7faa40149800 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa3f7fe640 10 monclient: dump: 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:57.192 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa459d1640 10 monclient: shutdown 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa459d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa459d1640 1 -- 172.21.15.33:0/4013712057 >> v1:172.21.15.153:6789/0 conn(0x7faa40149800 legacy=0x7faa40151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa459d1640 1 -- 172.21.15.33:0/4013712057 shutdown_connections 2024-03-29T11:40:57.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.188+0000 7faa459d1640 1 -- 172.21.15.33:0/4013712057 >> 172.21.15.33:0/4013712057 conn(0x7faa400b3a30 msgr2=0x7faa400b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.189+0000 7faa459d1640 1 -- 172.21.15.33:0/4013712057 shutdown_connections 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.189+0000 7faa459d1640 1 -- 172.21.15.33:0/4013712057 wait complete. 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.190+0000 7faa459d1640 10 monclient: build_initial_monmap 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.190+0000 7faa459d1640 10 monclient: monmap: 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:57.194 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.191+0000 7faa459d1640 1 librados: starting msgr at 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.191+0000 7faa459d1640 1 librados: starting objecter 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.191+0000 7faa459d1640 1 Processor -- start 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.191+0000 7faa459d1640 1 -- start start 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.191+0000 7faa459d1640 1 librados: setting wanted keys 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.191+0000 7faa459d1640 1 librados: calling monclient init 2024-03-29T11:40:57.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.191+0000 7faa459d1640 10 monclient: init 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 10 monclient(hunting): picked mon.noname-a con 0x7faa400aa500 addr v1:172.21.15.33:6789/0 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 10 monclient(hunting): picked mon.noname-b con 0x7faa40149800 addr v1:172.21.15.153:6789/0 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 10 monclient(hunting): picked mon.noname-c con 0x7faa4014a400 addr v1:172.21.15.33:6790/0 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4014bd00 con 0x7faa400aa500 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4014b0f0 con 0x7faa40149800 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4014bf50 con 0x7faa4014a400 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:57.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.192+0000 7faa459d1640 10 monclient(hunting): authenticate will time out at 1213.456310s 2024-03-29T11:40:57.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa449cf640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7faa400aa500 0x7faa40072100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49211/0 (socket says 172.21.15.33:49211) 2024-03-29T11:40:57.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa449cf640 1 -- 172.21.15.33:0/1537417958 learned_addr learned my addr 172.21.15.33:0/1537417958 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 10 client.?.objecter ms_handle_connect 0x7faa400aa500 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575277820 0 0) 0x7faa4014bd00 con 0x7faa400aa500 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:40:57.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.196+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa24004eb0 con 0x7faa400aa500 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.197+0000 7faa3d7fa640 10 client.?.objecter ms_handle_connect 0x7faa40149800 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.197+0000 7faa3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.197+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720430248 0 0) 0x7faa24004eb0 con 0x7faa400aa500 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa4014bd00 con 0x7faa400aa500 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7faa2c0018b0 con 0x7faa400aa500 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): dump: 2024-03-29T11:40:57.201 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427188766 0 0) 0x7faa4014b0f0 con 0x7faa40149800 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:57.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa24004eb0 con 0x7faa40149800 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2339930926 0 0) 0x7faa4014bd00 con 0x7faa400aa500 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 >> v1:172.21.15.33:6790/0 conn(0x7faa4014a400 legacy=0x7faa4006fef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 >> v1:172.21.15.153:6789/0 conn(0x7faa40149800 legacy=0x7faa4006f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 monclient: found mon.a 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.198+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4014c230 con 0x7faa400aa500 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa3d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa3d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:40:57.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 10 monclient: _renew_subs 2024-03-29T11:40:57.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:40:57.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4014acc0 con 0x7faa400aa500 2024-03-29T11:40:57.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:57.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 10 monclient: _renew_subs 2024-03-29T11:40:57.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:40:57.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa401e6de0 con 0x7faa400aa500 2024-03-29T11:40:57.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.199+0000 7faa459d1640 1 librados: init done 2024-03-29T11:40:57.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.200+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa2c001a80 con 0x7faa400aa500 2024-03-29T11:40:57.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.200+0000 7faa3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:57.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.200+0000 7faa3effd640 4 set_mon_vals no callback set 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.200+0000 7faa459d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.201+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7faa2c004a90 con 0x7faa400aa500 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.201+0000 7faa459d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.201+0000 7faa459d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.201+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa20005080 con 0x7faa400aa500 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.201+0000 7faa3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.201+0000 7faa3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.201+0000 7faa3d7fa640 10 monclient: dump: 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:40:57.206 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.202+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7faa2c01aa60 con 0x7faa400aa500 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.203+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7faa2c084a00 con 0x7faa400aa500 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.203+0000 7faa3ffff640 10 monclient: get_auth_request con 0x7faa2406dce0 auth_method 0 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.203+0000 7faa3d7fa640 10 client.4431.objecter ms_dispatch 0x7faa40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.203+0000 7faa3d7fa640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.203+0000 7faa3d7fa640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.203+0000 7faa3d7fa640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-29T11:40:57.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.203+0000 7faa3d7fa640 10 client.4431.objecter ms_handle_connect 0x7faa2406dce0 2024-03-29T11:40:57.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.211+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa2c050110 con 0x7faa400aa500 2024-03-29T11:40:57.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.211+0000 7faa3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:57.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.211+0000 7faa3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:57.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.532+0000 7faa459d1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:57.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.532+0000 7faa459d1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:57.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.532+0000 7faa459d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:40:57.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.532+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa20005330 con 0x7faa400aa500 2024-03-29T11:40:57.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.535+0000 7faa3d7fa640 1 -- 172.21.15.33:0/1537417958 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7faa2c052210 con 0x7faa400aa500 2024-03-29T11:40:57.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.535+0000 7faa3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:57.537 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.535+0000 7faa3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:57.537 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:40:57.537 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:57.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 10 librados: watch_flush enter 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 10 librados: watch_flush exit 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 10 client.4431.objecter successfully canceled tick 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 >> v1:172.21.15.153:6816/2737028074 conn(0x7faa2406dce0 legacy=0x7faa240701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 10 monclient: shutdown 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 >> v1:172.21.15.33:6789/0 conn(0x7faa400aa500 legacy=0x7faa40072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 shutdown_connections 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.538+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 >> 172.21.15.33:0/1537417958 conn(0x7faa400b3a30 msgr2=0x7faa40069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.539+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 shutdown_connections 2024-03-29T11:40:57.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.539+0000 7faa459d1640 1 -- 172.21.15.33:0/1537417958 wait complete. 2024-03-29T11:40:57.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:40:57.543+0000 7faa459d1640 1 librados: shutdown 2024-03-29T11:40:57.564 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:04.371 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.513+0000 7f5283650640 10 monclient: get_monmap_and_config 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.513+0000 7f5283650640 10 monclient: build_initial_monmap 2024-03-29T11:41:04.516 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.513+0000 7f5283650640 10 monclient: monmap: 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:04.514470+0000 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:04.514470+0000 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:04.517 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:04.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.515+0000 7f5283650640 1 Processor -- start 2024-03-29T11:41:04.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.515+0000 7f5283650640 1 -- start start 2024-03-29T11:41:04.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.515+0000 7f5283650640 10 monclient: init 2024-03-29T11:41:04.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:04.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 10 monclient(hunting): picked mon.noname-c con 0x7f527c0a9930 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 10 monclient(hunting): picked mon.noname-b con 0x7f527c149dc0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 10 monclient(hunting): picked mon.noname-a con 0x7f527c14ff40 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0aae60 con 0x7f527c14ff40 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0b4310 con 0x7f527c0a9930 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0afdd0 con 0x7f527c149dc0 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.516+0000 7f5283650640 10 monclient(hunting): authenticate will time out at 1220.779928s 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.517+0000 7f5281bed640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f527c14ff40 0x7f527c152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:51476/0 (socket says 172.21.15.33:51476) 2024-03-29T11:41:04.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.517+0000 7f5281bed640 1 -- 172.21.15.33:0/1660266713 learned_addr learned my addr 172.21.15.33:0/1660266713 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880357183 0 0) 0x7f527c0aae60 con 0x7f527c14ff40 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5268003520 con 0x7f527c14ff40 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3863565040 0 0) 0x7f5268003520 con 0x7f527c14ff40 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 >> v1:172.21.15.33:6790/0 conn(0x7f527c149dc0 legacy=0x7f527c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 >> v1:172.21.15.153:6789/0 conn(0x7f527c0a9930 legacy=0x7f527c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:04.524 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 monclient: found mon.noname-a 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c14a950 con 0x7f527c14ff40 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient: _finish_auth 0 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5283650640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5283650640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5278001670 con 0x7f527c14ff40 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5273fff640 10 monclient: dump: 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.525 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.520+0000 7f5283650640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52780018d0 con 0x7f527c14ff40 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5273fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5283650640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5283650640 4 set_mon_vals no callback set 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5273fff640 1 -- 172.21.15.33:0/1660266713 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5278001670 con 0x7f527c14ff40 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5273fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.526 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5273fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.521+0000 7f5273fff640 10 monclient: dump: 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.522+0000 7f5283650640 10 monclient: shutdown 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.522+0000 7f5283650640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.522+0000 7f5283650640 1 -- 172.21.15.33:0/1660266713 >> v1:172.21.15.33:6789/0 conn(0x7f527c14ff40 legacy=0x7f527c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.527 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.522+0000 7f5283650640 1 -- 172.21.15.33:0/1660266713 shutdown_connections 2024-03-29T11:41:04.528 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.522+0000 7f5283650640 1 -- 172.21.15.33:0/1660266713 >> 172.21.15.33:0/1660266713 conn(0x7f527c0b35e0 msgr2=0x7f527c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:04.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.523+0000 7f5283650640 1 -- 172.21.15.33:0/1660266713 shutdown_connections 2024-03-29T11:41:04.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.523+0000 7f5283650640 1 -- 172.21.15.33:0/1660266713 wait complete. 2024-03-29T11:41:04.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.524+0000 7f5283650640 10 monclient: build_initial_monmap 2024-03-29T11:41:04.530 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.524+0000 7f5283650640 10 monclient: monmap: 2024-03-29T11:41:04.530 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:04.530 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:04.531 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:04.531 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:04.531 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:04.531 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.524+0000 7f5283650640 1 librados: starting msgr at 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.524+0000 7f5283650640 1 librados: starting objecter 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.524+0000 7f5283650640 1 Processor -- start 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.526+0000 7f5283650640 1 -- start start 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.526+0000 7f5283650640 1 librados: setting wanted keys 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.526+0000 7f5283650640 1 librados: calling monclient init 2024-03-29T11:41:04.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.526+0000 7f5283650640 10 monclient: init 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 10 monclient(hunting): picked mon.noname-b con 0x7f527c14ff40 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 10 monclient(hunting): picked mon.noname-c con 0x7f527c13a330 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 10 monclient(hunting): picked mon.noname-a con 0x7f527c1df0a0 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5281bed640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f527c1df0a0 0x7f527c1df4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:57214/0 (socket says 172.21.15.33:57214) 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5281bed640 1 -- 172.21.15.33:0/2206709651 learned_addr learned my addr 172.21.15.33:0/2206709651 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c1dec10 con 0x7f527c1df0a0 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c144ce0 con 0x7f527c14ff40 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0ab1d0 con 0x7f527c13a330 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5283650640 10 monclient(hunting): authenticate will time out at 1220.791138s 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 10 client.?.objecter ms_handle_connect 0x7f527c1df0a0 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:04.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492621659 0 0) 0x7f527c1dec10 con 0x7f527c1df0a0 2024-03-29T11:41:04.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:04.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:04.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:04.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:41:04.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.527+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260004eb0 con 0x7f527c1df0a0 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.530+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 961194797 0 0) 0x7f5260004eb0 con 0x7f527c1df0a0 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f527c1dec10 con 0x7f527c1df0a0 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 client.?.objecter ms_handle_connect 0x7f527c14ff40 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5278001630 con 0x7f527c1df0a0 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:04.538 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865452686 0 0) 0x7f527c144ce0 con 0x7f527c14ff40 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:41:04.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260004eb0 con 0x7f527c14ff40 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 18394429 0 0) 0x7f527c1dec10 con 0x7f527c1df0a0 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 >> v1:172.21.15.33:6790/0 conn(0x7f527c13a330 legacy=0x7f527c13a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 >> v1:172.21.15.153:6789/0 conn(0x7f527c14ff40 legacy=0x7f527c13f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 monclient: found mon.a 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.531+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c144f50 con 0x7f527c1df0a0 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 10 monclient: _renew_subs 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:41:04.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c13f8d0 con 0x7f527c1df0a0 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 10 monclient: _renew_subs 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c1e7fc0 con 0x7f527c1df0a0 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5283650640 1 librados: init done 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5278004a90 con 0x7f527c1df0a0 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5278004e20 con 0x7f527c1df0a0 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5271ffb640 10 monclient: dump: 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.541 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:04.542 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:04.542 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:04.542 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:04.542 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:04.542 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:04.542 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:04.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.532+0000 7f5272ffd640 4 set_mon_vals no callback set 2024-03-29T11:41:04.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.535+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f5278004f80 con 0x7f527c1df0a0 2024-03-29T11:41:04.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.536+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f5278084bf0 con 0x7f527c1df0a0 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.536+0000 7f5271ffb640 10 client.4437.objecter ms_dispatch 0x7f527c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.536+0000 7f5271ffb640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.536+0000 7f5271ffb640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.536+0000 7f5271ffb640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.537+0000 7f52813ec640 10 monclient: get_auth_request con 0x7f526006dcc0 auth_method 0 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.537+0000 7f5283650640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.537+0000 7f5283650640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.537+0000 7f5283650640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.537+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c005080 con 0x7f527c1df0a0 2024-03-29T11:41:04.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.538+0000 7f5271ffb640 10 client.4437.objecter ms_handle_connect 0x7f526006dcc0 2024-03-29T11:41:04.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.546+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52780504d0 con 0x7f527c1df0a0 2024-03-29T11:41:04.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.546+0000 7f5271ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:04.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.546+0000 7f5271ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:04.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.870+0000 7f5283650640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:04.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.870+0000 7f5283650640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:04.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.870+0000 7f5283650640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:41:04.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.870+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f525c005330 con 0x7f527c1df0a0 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.873+0000 7f5271ffb640 1 -- 172.21.15.33:0/2206709651 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f5278052400 con 0x7f527c1df0a0 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.873+0000 7f5271ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:04.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.873+0000 7f5271ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:04.879 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:04.879 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:04.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.877+0000 7f5283650640 10 librados: watch_flush enter 2024-03-29T11:41:04.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.881+0000 7f5283650640 10 librados: watch_flush exit 2024-03-29T11:41:04.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.881+0000 7f5283650640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T11:41:04.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.881+0000 7f5283650640 10 client.4437.objecter successfully canceled tick 2024-03-29T11:41:04.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.881+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 >> v1:172.21.15.153:6816/2737028074 conn(0x7f526006dcc0 legacy=0x7f5260070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.881+0000 7f5283650640 10 monclient: shutdown 2024-03-29T11:41:04.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.881+0000 7f5283650640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:04.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.881+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 >> v1:172.21.15.33:6789/0 conn(0x7f527c1df0a0 legacy=0x7f527c1df4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:04.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.882+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 shutdown_connections 2024-03-29T11:41:04.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.882+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 >> 172.21.15.33:0/2206709651 conn(0x7f527c0b35e0 msgr2=0x7f527c140a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:04.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.882+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 shutdown_connections 2024-03-29T11:41:04.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.882+0000 7f5283650640 1 -- 172.21.15.33:0/2206709651 wait complete. 2024-03-29T11:41:04.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:04.886+0000 7f5283650640 1 librados: shutdown 2024-03-29T11:41:04.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:11.713 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.861+0000 7fa51a323640 10 monclient: get_monmap_and_config 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.861+0000 7fa51a323640 10 monclient: build_initial_monmap 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.861+0000 7fa51a323640 10 monclient: monmap: 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:11.863540+0000 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:11.863540+0000 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:11.865 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.863+0000 7fa51a323640 1 Processor -- start 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.863+0000 7fa51a323640 1 -- start start 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.863+0000 7fa51a323640 10 monclient: init 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.863+0000 7fa51a323640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.863+0000 7fa51a323640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.863+0000 7fa51a323640 10 monclient(hunting): picked mon.noname-a con 0x7fa5140b0500 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.863+0000 7fa51a323640 10 monclient(hunting): picked mon.noname-b con 0x7fa5140b08e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa51a323640 10 monclient(hunting): picked mon.noname-c con 0x7fa5140aeca0 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa51a323640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa514140620 con 0x7fa5140b0500 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa51a323640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51413daf0 con 0x7fa5140aeca0 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa51a323640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5140ae180 con 0x7fa5140b08e0 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa51a323640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:11.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa51a323640 10 monclient(hunting): authenticate will time out at 1228.128822s 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa5137fe640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fa5140b08e0 0x7fa51413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:43062/0 (socket says 172.21.15.33:43062) 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa5137fe640 1 -- 172.21.15.33:0/899879742 learned_addr learned my addr 172.21.15.33:0/899879742 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:11.868 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210046153 0 0) 0x7fa5140ae180 con 0x7fa5140b08e0 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa512ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa512ffd640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.864+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa500003520 con 0x7fa5140b08e0 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483844819 0 0) 0x7fa51413daf0 con 0x7fa5140aeca0 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5140ae180 con 0x7fa5140aeca0 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4018121942 0 0) 0x7fa500003520 con 0x7fa5140b08e0 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 >> v1:172.21.15.153:6789/0 conn(0x7fa5140aeca0 legacy=0x7fa51414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.869 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 >> v1:172.21.15.33:6789/0 conn(0x7fa5140b0500 legacy=0x7fa51413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 monclient: found mon.noname-b 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5140accf0 con 0x7fa5140b08e0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa4fc001690 con 0x7fa5140b08e0 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa51a323640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa51a323640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: dump: 2024-03-29T11:41:11.870 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa51a323640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4fc0017f0 con 0x7fa5140b08e0 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 1 -- 172.21.15.33:0/899879742 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa4fc001c00 con 0x7fa5140b08e0 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:11.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa512ffd640 10 monclient: dump: 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa51a323640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:11.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.865+0000 7fa51a323640 4 set_mon_vals no callback set 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.867+0000 7fa51a323640 10 monclient: shutdown 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.867+0000 7fa51a323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.867+0000 7fa51a323640 1 -- 172.21.15.33:0/899879742 >> v1:172.21.15.33:6790/0 conn(0x7fa5140b08e0 legacy=0x7fa51413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.867+0000 7fa51a323640 1 -- 172.21.15.33:0/899879742 shutdown_connections 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.867+0000 7fa51a323640 1 -- 172.21.15.33:0/899879742 >> 172.21.15.33:0/899879742 conn(0x7fa51406b660 msgr2=0x7fa51406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.867+0000 7fa51a323640 1 -- 172.21.15.33:0/899879742 shutdown_connections 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.867+0000 7fa51a323640 1 -- 172.21.15.33:0/899879742 wait complete. 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.868+0000 7fa51a323640 10 monclient: build_initial_monmap 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.868+0000 7fa51a323640 10 monclient: monmap: 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:11.877 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.868+0000 7fa51a323640 1 librados: starting msgr at 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.868+0000 7fa51a323640 1 librados: starting objecter 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.868+0000 7fa51a323640 1 Processor -- start 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.869+0000 7fa51a323640 1 -- start start 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.869+0000 7fa51a323640 1 librados: setting wanted keys 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.869+0000 7fa51a323640 1 librados: calling monclient init 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.869+0000 7fa51a323640 10 monclient: init 2024-03-29T11:41:11.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.869+0000 7fa51a323640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 10 monclient(hunting): picked mon.noname-a con 0x7fa5140aeca0 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 10 monclient(hunting): picked mon.noname-c con 0x7fa5140b0500 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 10 monclient(hunting): picked mon.noname-b con 0x7fa51413a9a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5141db770 con 0x7fa5140aeca0 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5141c86f0 con 0x7fa51413a9a0 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5141c8960 con 0x7fa5140b0500 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa51a323640 10 monclient(hunting): authenticate will time out at 1228.135210s 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa513fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fa5140aeca0 0x7fa5141daea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:54230/0 (socket says 172.21.15.33:54230) 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.870+0000 7fa513fff640 1 -- 172.21.15.33:0/1744226734 learned_addr learned my addr 172.21.15.33:0/1744226734 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 client.?.objecter ms_handle_connect 0x7fa51413a9a0 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 client.?.objecter ms_handle_connect 0x7fa5140b0500 2024-03-29T11:41:11.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 client.?.objecter ms_handle_connect 0x7fa5140aeca0 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418409273 0 0) 0x7fa5141c8960 con 0x7fa5140b0500 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f4004eb0 con 0x7fa5140b0500 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090668456 0 0) 0x7fa5141c86f0 con 0x7fa51413a9a0 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f4006230 con 0x7fa51413a9a0 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133411368 0 0) 0x7fa5141db770 con 0x7fa5140aeca0 2024-03-29T11:41:11.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f4006570 con 0x7fa5140aeca0 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936571719 0 0) 0x7fa4f4006230 con 0x7fa51413a9a0 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5141db770 con 0x7fa51413a9a0 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3115907723 0 0) 0x7fa4f4004eb0 con 0x7fa5140b0500 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f4006230 con 0x7fa5140b0500 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa50c0018b0 con 0x7fa51413a9a0 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): dump: 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.881 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa4fc001b00 con 0x7fa5140b0500 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.871+0000 7fa510ff9640 10 monclient(hunting): dump: 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.882 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3867013552 0 0) 0x7fa4f4006570 con 0x7fa5140aeca0 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f4004eb0 con 0x7fa5140aeca0 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa504001cf0 con 0x7fa5140aeca0 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient(hunting): dump: 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:11.883 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2629084313 0 0) 0x7fa5141db770 con 0x7fa51413a9a0 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 >> v1:172.21.15.33:6790/0 conn(0x7fa5140b0500 legacy=0x7fa5141cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 >> v1:172.21.15.33:6789/0 conn(0x7fa5140aeca0 legacy=0x7fa5141daea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 monclient: found mon.b 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:11.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5141cdae0 con 0x7fa51413a9a0 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient: _finish_auth 0 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 10 monclient: _renew_subs 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5141cdcd0 con 0x7fa51413a9a0 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 10 monclient: _renew_subs 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5141c6ed0 con 0x7fa51413a9a0 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa51a323640 1 librados: init done 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa50c001a10 con 0x7fa51413a9a0 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa510ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.872+0000 7fa5127fc640 4 set_mon_vals no callback set 2024-03-29T11:41:11.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.873+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa50c003790 con 0x7fa51413a9a0 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.873+0000 7fa510ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.874+0000 7fa510ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.874+0000 7fa510ff9640 10 monclient: dump: 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.874+0000 7fa51a323640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:11.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.874+0000 7fa51a323640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:11.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.874+0000 7fa51a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:11.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.874+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f8005080 con 0x7fa51413a9a0 2024-03-29T11:41:11.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.875+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fa50c019760 con 0x7fa51413a9a0 2024-03-29T11:41:11.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.876+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fa50c053260 con 0x7fa51413a9a0 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.876+0000 7fa510ff9640 10 client.4474.objecter ms_dispatch 0x7fa514000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.876+0000 7fa510ff9640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.876+0000 7fa510ff9640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.876+0000 7fa510ff9640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.876+0000 7fa513fff640 10 monclient: get_auth_request con 0x7fa4f4076e90 auth_method 0 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.876+0000 7fa510ff9640 10 client.4474.objecter ms_handle_connect 0x7fa4f4076e90 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.885+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa50c04f500 con 0x7fa51413a9a0 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.885+0000 7fa510ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:11.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:11.885+0000 7fa510ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:12.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.216+0000 7fa51a323640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:12.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.216+0000 7fa51a323640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:12.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.216+0000 7fa51a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:12.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.216+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4f8005330 con 0x7fa51413a9a0 2024-03-29T11:41:12.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.217+0000 7fa510ff9640 1 -- 172.21.15.33:0/1744226734 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fa50c04f6a0 con 0x7fa51413a9a0 2024-03-29T11:41:12.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.217+0000 7fa510ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:12.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.217+0000 7fa510ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:12.220 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:12.220 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:12.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.219+0000 7fa51a323640 10 librados: watch_flush enter 2024-03-29T11:41:12.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 10 librados: watch_flush exit 2024-03-29T11:41:12.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:41:12.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:41:12.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 >> v1:172.21.15.153:6816/2737028074 conn(0x7fa4f4076e90 legacy=0x7fa4f4079350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:12.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 10 monclient: shutdown 2024-03-29T11:41:12.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:12.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 >> v1:172.21.15.153:6789/0 conn(0x7fa51413a9a0 legacy=0x7fa5141c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:12.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 shutdown_connections 2024-03-29T11:41:12.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 >> 172.21.15.33:0/1744226734 conn(0x7fa51406b660 msgr2=0x7fa5140a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:12.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 shutdown_connections 2024-03-29T11:41:12.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 1 -- 172.21.15.33:0/1744226734 wait complete. 2024-03-29T11:41:12.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:12.220+0000 7fa51a323640 1 librados: shutdown 2024-03-29T11:41:12.239 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:19.046 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.193+0000 7fe1dd183640 10 monclient: get_monmap_and_config 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.193+0000 7fe1dd183640 10 monclient: build_initial_monmap 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.193+0000 7fe1dd183640 10 monclient: monmap: 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:19.194830+0000 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:19.194830+0000 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:19.196 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:19.197 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:19.197 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:19.197 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:19.197 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:19.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.194+0000 7fe1dd183640 1 Processor -- start 2024-03-29T11:41:19.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.194+0000 7fe1dd183640 1 -- start start 2024-03-29T11:41:19.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.194+0000 7fe1dd183640 10 monclient: init 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d81457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d813f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d80a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d80aaef0 con 0x7fe1d81457f0 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d8145bd0 con 0x7fe1d813f650 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d8145eb0 con 0x7fe1d80a9d70 2024-03-29T11:41:19.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.195+0000 7fe1dd183640 10 monclient(hunting): authenticate will time out at 1235.460574s 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d759e640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fe1d80a9d70 0x7fe1d814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:41136/0 (socket says 172.21.15.33:41136) 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d759e640 1 -- 172.21.15.33:0/2597529001 learned_addr learned my addr 172.21.15.33:0/2597529001 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061192775 0 0) 0x7fe1d8145eb0 con 0x7fe1d80a9d70 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d5d9b640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:41:19.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.196+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c0003520 con 0x7fe1d80a9d70 2024-03-29T11:41:19.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009395784 0 0) 0x7fe1c0003520 con 0x7fe1d80a9d70 2024-03-29T11:41:19.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 >> v1:172.21.15.153:6789/0 conn(0x7fe1d813f650 legacy=0x7fe1d80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 >> v1:172.21.15.33:6789/0 conn(0x7fe1d81457f0 legacy=0x7fe1d813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 1 monclient: found mon.noname-b 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8146340 con 0x7fe1d80a9d70 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe1d0001a10 con 0x7fe1d80a9d70 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1d5d9b640 10 monclient: dump: 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.202 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1dd183640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.197+0000 7fe1dd183640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1d0001b70 con 0x7fe1d80a9d70 2024-03-29T11:41:19.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1d5d9b640 1 -- 172.21.15.33:0/2597529001 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe1d0001fc0 con 0x7fe1d80a9d70 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1dd183640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1dd183640 4 set_mon_vals no callback set 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1d5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.198+0000 7fe1d5d9b640 10 monclient: dump: 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:19.204 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:19.205 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:19.205 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:19.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.199+0000 7fe1dd183640 10 monclient: shutdown 2024-03-29T11:41:19.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.199+0000 7fe1dd183640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:19.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.199+0000 7fe1dd183640 1 -- 172.21.15.33:0/2597529001 >> v1:172.21.15.33:6790/0 conn(0x7fe1d80a9d70 legacy=0x7fe1d814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.199+0000 7fe1dd183640 1 -- 172.21.15.33:0/2597529001 shutdown_connections 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.199+0000 7fe1dd183640 1 -- 172.21.15.33:0/2597529001 >> 172.21.15.33:0/2597529001 conn(0x7fe1d8066a00 msgr2=0x7fe1d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.199+0000 7fe1dd183640 1 -- 172.21.15.33:0/2597529001 shutdown_connections 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.199+0000 7fe1dd183640 1 -- 172.21.15.33:0/2597529001 wait complete. 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.200+0000 7fe1dd183640 10 monclient: build_initial_monmap 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.200+0000 7fe1dd183640 10 monclient: monmap: 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:19.213 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.201+0000 7fe1dd183640 1 librados: starting msgr at 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.201+0000 7fe1dd183640 1 librados: starting objecter 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.201+0000 7fe1dd183640 1 Processor -- start 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.201+0000 7fe1dd183640 1 -- start start 2024-03-29T11:41:19.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.201+0000 7fe1dd183640 1 librados: setting wanted keys 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.201+0000 7fe1dd183640 1 librados: calling monclient init 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.201+0000 7fe1dd183640 10 monclient: init 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.202+0000 7fe1dd183640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.202+0000 7fe1dd183640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d80a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d813f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d81457f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d81db4e0 con 0x7fe1d80a9d70 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d81db970 con 0x7fe1d813f650 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d81dbb00 con 0x7fe1d81457f0 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1dd183640 10 monclient(hunting): authenticate will time out at 1235.467832s 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1d759e640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fe1d81457f0 0x7fe1d81e21c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:41372/0 (socket says 172.21.15.33:41372) 2024-03-29T11:41:19.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1d759e640 1 -- 172.21.15.33:0/165512971 learned_addr learned my addr 172.21.15.33:0/165512971 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 client.?.objecter ms_handle_connect 0x7fe1d81457f0 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825090211 0 0) 0x7fe1d81dbb00 con 0x7fe1d81457f0 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4004eb0 con 0x7fe1d81457f0 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 client.?.objecter ms_handle_connect 0x7fe1d813f650 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 client.?.objecter ms_handle_connect 0x7fe1d80a9d70 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:19.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4285228110 0 0) 0x7fe1c4004eb0 con 0x7fe1d81457f0 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d81dbb00 con 0x7fe1d81457f0 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe1d0001e80 con 0x7fe1d81457f0 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.203+0000 7fe1bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:19.217 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643888286 0 0) 0x7fe1d81db970 con 0x7fe1d813f650 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4004eb0 con 0x7fe1d813f650 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496353278 0 0) 0x7fe1d81db4e0 con 0x7fe1d80a9d70 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d81db970 con 0x7fe1d80a9d70 2024-03-29T11:41:19.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1942860058 0 0) 0x7fe1d81dbb00 con 0x7fe1d81457f0 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 >> v1:172.21.15.153:6789/0 conn(0x7fe1d813f650 legacy=0x7fe1d81d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.204+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 >> v1:172.21.15.33:6789/0 conn(0x7fe1d80a9d70 legacy=0x7fe1d81dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 1 monclient: found mon.c 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d81d7e90 con 0x7fe1d81457f0 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 10 monclient: _renew_subs 2024-03-29T11:41:19.219 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d80b0cd0 con 0x7fe1d81457f0 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 10 monclient: _renew_subs 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d80b1020 con 0x7fe1d81457f0 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1dd183640 1 librados: init done 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1d00020a0 con 0x7fe1d81457f0 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe1d0005110 con 0x7fe1d81457f0 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.205+0000 7fe1bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.206+0000 7fe1d559a640 4 set_mon_vals no callback set 2024-03-29T11:41:19.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.206+0000 7fe1bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.206+0000 7fe1bf7fe640 10 monclient: dump: 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:19.221 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:19.222 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:19.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.206+0000 7fe1dd183640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:19.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.207+0000 7fe1dd183640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:19.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.207+0000 7fe1dd183640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:19.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.207+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b8005080 con 0x7fe1d81457f0 2024-03-29T11:41:19.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.209+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fe1d001b0e0 con 0x7fe1d81457f0 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.210+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fe1d0084f80 con 0x7fe1d81457f0 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.210+0000 7fe1bf7fe640 10 client.4499.objecter ms_dispatch 0x7fe1d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.210+0000 7fe1bf7fe640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.210+0000 7fe1bf7fe640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.210+0000 7fe1bf7fe640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.210+0000 7fe1d6d9d640 10 monclient: get_auth_request con 0x7fe1c406dc30 auth_method 0 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.211+0000 7fe1bf7fe640 10 client.4499.objecter ms_handle_connect 0x7fe1c406dc30 2024-03-29T11:41:19.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.219+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1d0050860 con 0x7fe1d81457f0 2024-03-29T11:41:19.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.219+0000 7fe1bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:19.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.219+0000 7fe1bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:19.621 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.618+0000 7fe1dd183640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:19.621 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.618+0000 7fe1dd183640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:19.621 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.618+0000 7fe1dd183640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:19.621 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.618+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1b8005330 con 0x7fe1d81457f0 2024-03-29T11:41:19.622 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.619+0000 7fe1bf7fe640 1 -- 172.21.15.33:0/165512971 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fe1d0052790 con 0x7fe1d81457f0 2024-03-29T11:41:19.622 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.619+0000 7fe1bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:19.622 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.619+0000 7fe1bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:19.622 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:19.622 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 10 librados: watch_flush enter 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 10 librados: watch_flush exit 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 >> v1:172.21.15.153:6816/2737028074 conn(0x7fe1c406dc30 legacy=0x7fe1c40700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 10 monclient: shutdown 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:19.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 >> v1:172.21.15.33:6790/0 conn(0x7fe1d81457f0 legacy=0x7fe1d81e21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:19.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 shutdown_connections 2024-03-29T11:41:19.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.622+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 >> 172.21.15.33:0/165512971 conn(0x7fe1d8066a00 msgr2=0x7fe1d813ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:19.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.623+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 shutdown_connections 2024-03-29T11:41:19.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.623+0000 7fe1dd183640 1 -- 172.21.15.33:0/165512971 wait complete. 2024-03-29T11:41:19.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:19.623+0000 7fe1dd183640 1 librados: shutdown 2024-03-29T11:41:19.640 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:26.447 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:26.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.592+0000 7f1cc717b640 10 monclient: get_monmap_and_config 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.592+0000 7f1cc717b640 10 monclient: build_initial_monmap 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.592+0000 7f1cc717b640 10 monclient: monmap: 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:26.594212+0000 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:26.594212+0000 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:26.596 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:26.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.593+0000 7f1cc717b640 1 Processor -- start 2024-03-29T11:41:26.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.593+0000 7f1cc717b640 1 -- start start 2024-03-29T11:41:26.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.593+0000 7f1cc717b640 10 monclient: init 2024-03-29T11:41:26.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:26.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:26.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc00b1f60 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:26.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc0071eb0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:26.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc0143d10 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:26.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc006ef80 con 0x7f1cc0143d10 2024-03-29T11:41:26.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc006f260 con 0x7f1cc00b1f60 2024-03-29T11:41:26.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc00b40c0 con 0x7f1cc0071eb0 2024-03-29T11:41:26.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:26.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.594+0000 7f1cc717b640 10 monclient(hunting): authenticate will time out at 1242.859612s 2024-03-29T11:41:26.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.598+0000 7f1cc5718640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f1cc0143d10 0x7f1cc0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:53463/0 (socket says 172.21.15.33:53463) 2024-03-29T11:41:26.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.598+0000 7f1cc5718640 1 -- 172.21.15.33:0/1529788475 learned_addr learned my addr 172.21.15.33:0/1529788475 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:26.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.601+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191409158 0 0) 0x7f1cc006f260 con 0x7f1cc00b1f60 2024-03-29T11:41:26.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.601+0000 7f1cb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:26.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.601+0000 7f1cb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:26.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.601+0000 7f1cb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:26.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.601+0000 7f1cb77fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:41:26.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.601+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca8003520 con 0x7f1cc00b1f60 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3756975375 0 0) 0x7f1ca8003520 con 0x7f1cc00b1f60 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 >> v1:172.21.15.33:6790/0 conn(0x7f1cc0071eb0 legacy=0x7f1cc0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 >> v1:172.21.15.33:6789/0 conn(0x7f1cc0143d10 legacy=0x7f1cc0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 1 monclient: found mon.noname-c 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc0155d70 con 0x7f1cc00b1f60 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:26.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:26.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1cbc001690 con 0x7f1cc00b1f60 2024-03-29T11:41:26.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:26.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cb77fe640 10 monclient: dump: 2024-03-29T11:41:26.606 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:26.606 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cc717b640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:41:26.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.602+0000 7f1cc717b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:26.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.605+0000 7f1cb77fe640 1 -- 172.21.15.33:0/1529788475 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cbc001860 con 0x7f1cc00b1f60 2024-03-29T11:41:26.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.605+0000 7f1cb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:26.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.605+0000 7f1cc717b640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:26.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.605+0000 7f1cc717b640 4 set_mon_vals no callback set 2024-03-29T11:41:26.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.606+0000 7f1cc717b640 10 monclient: shutdown 2024-03-29T11:41:26.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.606+0000 7f1cc717b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:26.610 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.606+0000 7f1cc717b640 1 -- 172.21.15.33:0/1529788475 >> v1:172.21.15.153:6789/0 conn(0x7f1cc00b1f60 legacy=0x7f1cc0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.610 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.606+0000 7f1cc717b640 1 -- 172.21.15.33:0/1529788475 shutdown_connections 2024-03-29T11:41:26.610 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.606+0000 7f1cc717b640 1 -- 172.21.15.33:0/1529788475 >> 172.21.15.33:0/1529788475 conn(0x7f1cc0066a00 msgr2=0x7f1cc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:26.610 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.608+0000 7f1cc717b640 1 -- 172.21.15.33:0/1529788475 shutdown_connections 2024-03-29T11:41:26.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.608+0000 7f1cc717b640 1 -- 172.21.15.33:0/1529788475 wait complete. 2024-03-29T11:41:26.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.609+0000 7f1cc717b640 10 monclient: build_initial_monmap 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.609+0000 7f1cc717b640 10 monclient: monmap: 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:26.613 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.609+0000 7f1cc717b640 1 librados: starting msgr at 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.609+0000 7f1cc717b640 1 librados: starting objecter 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.609+0000 7f1cc717b640 1 Processor -- start 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.610+0000 7f1cc717b640 1 -- start start 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.610+0000 7f1cc717b640 1 librados: setting wanted keys 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.610+0000 7f1cc717b640 1 librados: calling monclient init 2024-03-29T11:41:26.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.610+0000 7f1cc717b640 10 monclient: init 2024-03-29T11:41:26.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.611+0000 7f1cc717b640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:26.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.611+0000 7f1cc717b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc0143d10 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc0153ac0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc01e3c80 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc013b2c0 con 0x7f1cc0143d10 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc013b5a0 con 0x7f1cc01e3c80 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc013b880 con 0x7f1cc0153ac0 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc717b640 10 monclient(hunting): authenticate will time out at 1242.879302s 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc4f17640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f1cc0143d10 0x7f1cc0139fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:53933/0 (socket says 172.21.15.33:53933) 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.614+0000 7f1cc4f17640 1 -- 172.21.15.33:0/2595327844 learned_addr learned my addr 172.21.15.33:0/2595327844 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:26.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 client.?.objecter ms_handle_connect 0x7f1cc0143d10 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 client.?.objecter ms_handle_connect 0x7f1cc01e3c80 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671093839 0 0) 0x7f1cc013b2c0 con 0x7f1cc0143d10 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cac004eb0 con 0x7f1cc0143d10 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852099315 0 0) 0x7f1cc013b5a0 con 0x7f1cc01e3c80 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:41:26.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.615+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc013b2c0 con 0x7f1cc01e3c80 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.616+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1734610142 0 0) 0x7f1cc013b2c0 con 0x7f1cc01e3c80 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.616+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cc013b5a0 con 0x7f1cc01e3c80 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.616+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506168818 0 0) 0x7f1cac004eb0 con 0x7f1cc0143d10 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.616+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cc013b2c0 con 0x7f1cc0143d10 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.617+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1cb8001a80 con 0x7f1cc01e3c80 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.617+0000 7f1cb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.617+0000 7f1cb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.617+0000 7f1cb57fa640 10 monclient(hunting): dump: 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:26.626 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.617+0000 7f1cb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 client.?.objecter ms_handle_connect 0x7f1cc0153ac0 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1cbc0035f0 con 0x7f1cc0143d10 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 monclient(hunting): dump: 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.627 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884265128 0 0) 0x7f1cc013b880 con 0x7f1cc0153ac0 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cac004eb0 con 0x7f1cc0153ac0 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 684763472 0 0) 0x7f1cc013b5a0 con 0x7f1cc01e3c80 2024-03-29T11:41:26.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.618+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 >> v1:172.21.15.33:6790/0 conn(0x7f1cc0153ac0 legacy=0x7f1cc0153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 >> v1:172.21.15.33:6789/0 conn(0x7f1cc0143d10 legacy=0x7f1cc0139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 1 monclient: found mon.b 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc013bb60 con 0x7f1cc01e3c80 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 10 monclient: _renew_subs 2024-03-29T11:41:26.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc013a850 con 0x7f1cc01e3c80 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 10 monclient: _renew_subs 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc01e7790 con 0x7f1cc01e3c80 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.619+0000 7f1cc717b640 1 librados: init done 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.620+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cb8001f70 con 0x7f1cc01e3c80 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.620+0000 7f1cb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.620+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1cb8004d00 con 0x7f1cc01e3c80 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.620+0000 7f1cb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.620+0000 7f1cb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.620+0000 7f1cb57fa640 10 monclient: dump: 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:26.630 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:26.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.620+0000 7f1cb6ffd640 4 set_mon_vals no callback set 2024-03-29T11:41:26.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.623+0000 7f1cc717b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:26.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.623+0000 7f1cc717b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:26.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.623+0000 7f1cc717b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:26.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.623+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca0005080 con 0x7f1cc01e3c80 2024-03-29T11:41:26.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.623+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f1cb8004e60 con 0x7f1cc01e3c80 2024-03-29T11:41:26.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.624+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f1cb8084a50 con 0x7f1cc01e3c80 2024-03-29T11:41:26.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.624+0000 7f1cc4f17640 10 monclient: get_auth_request con 0x7f1cac06df90 auth_method 0 2024-03-29T11:41:26.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.624+0000 7f1cb57fa640 10 client.4486.objecter ms_dispatch 0x7f1cc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:26.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.624+0000 7f1cb57fa640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:26.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.624+0000 7f1cb57fa640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:26.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.624+0000 7f1cb57fa640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:41:26.633 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.624+0000 7f1cb57fa640 10 client.4486.objecter ms_handle_connect 0x7f1cac06df90 2024-03-29T11:41:26.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.633+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cb8050330 con 0x7f1cc01e3c80 2024-03-29T11:41:26.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.633+0000 7f1cb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:26.636 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.633+0000 7f1cb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:26.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.961+0000 7f1cc717b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:26.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.961+0000 7f1cc717b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:26.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.961+0000 7f1cc717b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:26.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.961+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ca0005330 con 0x7f1cc01e3c80 2024-03-29T11:41:26.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.963+0000 7f1cb57fa640 1 -- 172.21.15.33:0/2595327844 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f1cb8052260 con 0x7f1cc01e3c80 2024-03-29T11:41:26.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.963+0000 7f1cb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:26.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.963+0000 7f1cb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:26.967 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:26.967 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:26.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.965+0000 7f1cc717b640 10 librados: watch_flush enter 2024-03-29T11:41:26.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.966+0000 7f1cc717b640 10 librados: watch_flush exit 2024-03-29T11:41:26.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.966+0000 7f1cc717b640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:41:26.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.966+0000 7f1cc717b640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:41:26.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.966+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 >> v1:172.21.15.153:6816/2737028074 conn(0x7f1cac06df90 legacy=0x7f1cac070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.966+0000 7f1cc717b640 10 monclient: shutdown 2024-03-29T11:41:26.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.966+0000 7f1cc717b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:26.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.966+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 >> v1:172.21.15.153:6789/0 conn(0x7f1cc01e3c80 legacy=0x7f1cc01e6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:26.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.967+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 shutdown_connections 2024-03-29T11:41:26.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.967+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 >> 172.21.15.33:0/2595327844 conn(0x7f1cc0066a00 msgr2=0x7f1cc0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:26.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.968+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 shutdown_connections 2024-03-29T11:41:26.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.968+0000 7f1cc717b640 1 -- 172.21.15.33:0/2595327844 wait complete. 2024-03-29T11:41:26.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:26.969+0000 7f1cc717b640 1 librados: shutdown 2024-03-29T11:41:26.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:33.792 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:33.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.935+0000 7fad7c80f640 10 monclient: get_monmap_and_config 2024-03-29T11:41:33.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.935+0000 7fad7c80f640 10 monclient: build_initial_monmap 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.935+0000 7fad7c80f640 10 monclient: monmap: 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:33.936002+0000 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:33.936002+0000 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:33.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.936+0000 7fad7c80f640 1 Processor -- start 2024-03-29T11:41:33.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.937+0000 7fad7c80f640 1 -- start start 2024-03-29T11:41:33.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.937+0000 7fad7c80f640 10 monclient: init 2024-03-29T11:41:33.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.938+0000 7fad7c80f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:33.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.938+0000 7fad7c80f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:33.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.938+0000 7fad7c80f640 10 monclient(hunting): picked mon.noname-a con 0x7fad780a9750 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:33.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.938+0000 7fad7c80f640 10 monclient(hunting): picked mon.noname-c con 0x7fad78149ac0 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:33.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.938+0000 7fad6ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fad78149ac0 0x7fad78149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:49424/0 (socket says 172.21.15.33:49424) 2024-03-29T11:41:33.941 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.939+0000 7fad7c80f640 10 monclient(hunting): picked mon.noname-b con 0x7fad7814fe50 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:33.941 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.939+0000 7fad7c80f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780aa4a0 con 0x7fad780a9750 2024-03-29T11:41:33.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.939+0000 7fad7c80f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7814a680 con 0x7fad78149ac0 2024-03-29T11:41:33.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.939+0000 7fad7c80f640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780af290 con 0x7fad7814fe50 2024-03-29T11:41:33.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.939+0000 7fad7c80f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:33.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.939+0000 7fad7c80f640 10 monclient(hunting): authenticate will time out at 1250.203502s 2024-03-29T11:41:33.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.939+0000 7fad6ffff640 1 -- 172.21.15.33:0/1970016266 learned_addr learned my addr 172.21.15.33:0/1970016266 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:33.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.940+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085812212 0 0) 0x7fad7814a680 con 0x7fad78149ac0 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad64003520 con 0x7fad78149ac0 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877414333 0 0) 0x7fad780af290 con 0x7fad7814fe50 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:41:33.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad7814a680 con 0x7fad7814fe50 2024-03-29T11:41:33.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336831668 0 0) 0x7fad7814a680 con 0x7fad7814fe50 2024-03-29T11:41:33.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 >> v1:172.21.15.153:6789/0 conn(0x7fad78149ac0 legacy=0x7fad78149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:33.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.941+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 >> v1:172.21.15.33:6789/0 conn(0x7fad780a9750 legacy=0x7fad781436f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 1 monclient: found mon.noname-b 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780afb40 con 0x7fad7814fe50 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 10 monclient: _finish_auth 0 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.942+0000 7fad75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.943+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fad700019f0 con 0x7fad7814fe50 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.943+0000 7fad75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.943+0000 7fad75d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.943+0000 7fad75d9b640 10 monclient: dump: 2024-03-29T11:41:33.947 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.943+0000 7fad7c80f640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.943+0000 7fad7c80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad70001b50 con 0x7fad7814fe50 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad75d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad75d9b640 1 -- 172.21.15.33:0/1970016266 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fad70001fa0 con 0x7fad7814fe50 2024-03-29T11:41:33.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad75d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad75d9b640 10 monclient: dump: 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad7c80f640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad7c80f640 4 set_mon_vals no callback set 2024-03-29T11:41:33.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad7c80f640 10 monclient: shutdown 2024-03-29T11:41:33.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad7c80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:33.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.944+0000 7fad7c80f640 1 -- 172.21.15.33:0/1970016266 >> v1:172.21.15.33:6790/0 conn(0x7fad7814fe50 legacy=0x7fad78152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:33.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.945+0000 7fad7c80f640 1 -- 172.21.15.33:0/1970016266 shutdown_connections 2024-03-29T11:41:33.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.945+0000 7fad7c80f640 1 -- 172.21.15.33:0/1970016266 >> 172.21.15.33:0/1970016266 conn(0x7fad780b35e0 msgr2=0x7fad780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:33.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.945+0000 7fad7c80f640 1 -- 172.21.15.33:0/1970016266 shutdown_connections 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.945+0000 7fad7c80f640 1 -- 172.21.15.33:0/1970016266 wait complete. 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 10 monclient: build_initial_monmap 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 10 monclient: monmap: 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 1 librados: starting msgr at 2024-03-29T11:41:33.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 1 librados: starting objecter 2024-03-29T11:41:33.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 1 Processor -- start 2024-03-29T11:41:33.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 1 -- start start 2024-03-29T11:41:33.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 1 librados: setting wanted keys 2024-03-29T11:41:33.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 1 librados: calling monclient init 2024-03-29T11:41:33.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.947+0000 7fad7c80f640 10 monclient: init 2024-03-29T11:41:33.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.948+0000 7fad7c80f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:33.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.948+0000 7fad7c80f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:33.954 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.948+0000 7fad7c80f640 10 monclient(hunting): picked mon.noname-c con 0x7fad780a9750 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.950+0000 7fad7c80f640 10 monclient(hunting): picked mon.noname-b con 0x7fad78149ac0 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.950+0000 7fad7c80f640 10 monclient(hunting): picked mon.noname-a con 0x7fad7814fe50 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.950+0000 7fad7c80f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad781db5e0 con 0x7fad7814fe50 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.950+0000 7fad7c80f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780ad4f0 con 0x7fad78149ac0 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.950+0000 7fad7c80f640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad781dc4b0 con 0x7fad780a9750 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.950+0000 7fad7c80f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.950+0000 7fad7c80f640 10 monclient(hunting): authenticate will time out at 1250.214328s 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.952+0000 7fad6ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fad78149ac0 0x7fad7813c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:49570/0 (socket says 172.21.15.33:49570) 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.952+0000 7fad6ffff640 1 -- 172.21.15.33:0/1765042103 learned_addr learned my addr 172.21.15.33:0/1765042103 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:33.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.952+0000 7fad7659c640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fad780a9750 0x7fad781dbda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:38681/0 (socket says 172.21.15.33:38681) 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 client.?.objecter ms_handle_connect 0x7fad78149ac0 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661829728 0 0) 0x7fad780ad4f0 con 0x7fad78149ac0 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:41:33.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad58004eb0 con 0x7fad78149ac0 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4132194954 0 0) 0x7fad58004eb0 con 0x7fad78149ac0 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad780ad4f0 con 0x7fad78149ac0 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fad68001a70 con 0x7fad78149ac0 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): dump: 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:33.961 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.953+0000 7fad6effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 client.?.objecter ms_handle_connect 0x7fad7814fe50 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871020427 0 0) 0x7fad781db5e0 con 0x7fad7814fe50 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad58004eb0 con 0x7fad7814fe50 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1630470066 0 0) 0x7fad58004eb0 con 0x7fad7814fe50 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad781db5e0 con 0x7fad7814fe50 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fad70001e60 con 0x7fad7814fe50 2024-03-29T11:41:33.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient(hunting): dump: 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3716108700 0 0) 0x7fad780ad4f0 con 0x7fad78149ac0 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 >> v1:172.21.15.33:6790/0 conn(0x7fad780a9750 legacy=0x7fad781dbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:33.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 >> v1:172.21.15.33:6789/0 conn(0x7fad7814fe50 legacy=0x7fad7813a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 monclient: found mon.b 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad781db880 con 0x7fad78149ac0 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient: _finish_auth 0 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 10 monclient: _renew_subs 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7813c900 con 0x7fad78149ac0 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 10 monclient: _renew_subs 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:33.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.954+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7813ac20 con 0x7fad78149ac0 2024-03-29T11:41:33.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.955+0000 7fad7c80f640 1 librados: init done 2024-03-29T11:41:33.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.956+0000 7fad7c80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:33.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.956+0000 7fad7c80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:33.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.956+0000 7fad7c80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:33.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.956+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad78075f10 con 0x7fad78149ac0 2024-03-29T11:41:33.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.958+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad68001cb0 con 0x7fad78149ac0 2024-03-29T11:41:33.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.958+0000 7fad6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:33.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.958+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fad68004c30 con 0x7fad78149ac0 2024-03-29T11:41:33.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.958+0000 7fad6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:33.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.958+0000 7fad6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:33.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.958+0000 7fad6effd640 10 monclient: dump: 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.959+0000 7fad7559a640 4 set_mon_vals no callback set 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.959+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fad6801ac00 con 0x7fad78149ac0 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.960+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fad68084b60 con 0x7fad78149ac0 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.960+0000 7fad6effd640 10 client.4492.objecter ms_dispatch 0x7fad78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.960+0000 7fad6effd640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:33.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.960+0000 7fad6effd640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:33.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.961+0000 7fad6effd640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T11:41:33.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.962+0000 7fad7659c640 10 monclient: get_auth_request con 0x7fad5807eeb0 auth_method 0 2024-03-29T11:41:33.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.962+0000 7fad6effd640 10 client.4492.objecter ms_handle_connect 0x7fad5807eeb0 2024-03-29T11:41:33.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.964+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad68050440 con 0x7fad78149ac0 2024-03-29T11:41:33.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.964+0000 7fad6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:33.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:33.964+0000 7fad6effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:34.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.308+0000 7fad7c80f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:34.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.308+0000 7fad7c80f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:34.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.308+0000 7fad7c80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:41:34.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.308+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad7814a9f0 con 0x7fad78149ac0 2024-03-29T11:41:34.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.312+0000 7fad6effd640 1 -- 172.21.15.33:0/1765042103 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fad68052370 con 0x7fad78149ac0 2024-03-29T11:41:34.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.312+0000 7fad6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:34.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.312+0000 7fad6effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:34.316 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:34.316 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:34.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 10 librados: watch_flush enter 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 10 librados: watch_flush exit 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 10 client.4492.objecter successfully canceled tick 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 >> v1:172.21.15.153:6816/2737028074 conn(0x7fad5807eeb0 legacy=0x7fad58081370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 10 monclient: shutdown 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 >> v1:172.21.15.153:6789/0 conn(0x7fad78149ac0 legacy=0x7fad7813c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 shutdown_connections 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 >> 172.21.15.33:0/1765042103 conn(0x7fad780b35e0 msgr2=0x7fad780b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 shutdown_connections 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.316+0000 7fad7c80f640 1 -- 172.21.15.33:0/1765042103 wait complete. 2024-03-29T11:41:34.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:34.317+0000 7fad7c80f640 1 librados: shutdown 2024-03-29T11:41:34.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:41.143 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:41.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.294+0000 7f22c6596640 10 monclient: get_monmap_and_config 2024-03-29T11:41:41.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.294+0000 7f22c6596640 10 monclient: build_initial_monmap 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.294+0000 7f22c6596640 10 monclient: monmap: 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:41.294924+0000 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:41.294924+0000 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:41.297 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:41.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.295+0000 7f22c6596640 1 Processor -- start 2024-03-29T11:41:41.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.295+0000 7f22c6596640 1 -- start start 2024-03-29T11:41:41.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.295+0000 7f22c6596640 10 monclient: init 2024-03-29T11:41:41.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.296+0000 7f22c6596640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:41.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.296+0000 7f22c6596640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:41.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.296+0000 7f22c6596640 10 monclient(hunting): picked mon.noname-b con 0x7f22b80068b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:41.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.297+0000 7f22c6596640 10 monclient(hunting): picked mon.noname-a con 0x7f22b8006c90 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:41.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.297+0000 7f22c6596640 10 monclient(hunting): picked mon.noname-c con 0x7f22b802b4e0 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:41.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.297+0000 7f22c6596640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b80c75f0 con 0x7f22b8006c90 2024-03-29T11:41:41.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.297+0000 7f22c6596640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b802c850 con 0x7f22b802b4e0 2024-03-29T11:41:41.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.297+0000 7f22c6596640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b802afc0 con 0x7f22b80068b0 2024-03-29T11:41:41.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.297+0000 7f22c6596640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:41.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.297+0000 7f22c6596640 10 monclient(hunting): authenticate will time out at 1257.561006s 2024-03-29T11:41:41.303 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.301+0000 7f22bf7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f22b8006c90 0x7f22b80cc0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:57741/0 (socket says 172.21.15.33:57741) 2024-03-29T11:41:41.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.301+0000 7f22bf7fe640 1 -- 172.21.15.33:0/548281164 learned_addr learned my addr 172.21.15.33:0/548281164 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:41.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465777768 0 0) 0x7f22b80c75f0 con 0x7f22b8006c90 2024-03-29T11:41:41.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b0003520 con 0x7f22b8006c90 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290199270 0 0) 0x7f22b802afc0 con 0x7f22b80068b0 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:41:41.306 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.303+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b80c75f0 con 0x7f22b80068b0 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400400781 0 0) 0x7f22b0003520 con 0x7f22b8006c90 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 >> v1:172.21.15.33:6790/0 conn(0x7f22b80068b0 legacy=0x7f22b80c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 >> v1:172.21.15.153:6789/0 conn(0x7f22b802b4e0 legacy=0x7f22b80cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 1 monclient: found mon.noname-a 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b802abc0 con 0x7f22b8006c90 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f22ac001690 con 0x7f22b8006c90 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:41.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22beffd640 10 monclient: dump: 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22c6596640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:41:41.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.304+0000 7f22c6596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22ac0018f0 con 0x7f22b8006c90 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22beffd640 1 -- 172.21.15.33:0/548281164 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f22ac001d00 con 0x7f22b8006c90 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22beffd640 10 monclient: dump: 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.310 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.311 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:41.311 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:41.311 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:41.311 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:41.311 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:41.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22c6596640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:41.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.308+0000 7f22c6596640 4 set_mon_vals no callback set 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.309+0000 7f22c6596640 10 monclient: shutdown 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.309+0000 7f22c6596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.309+0000 7f22c6596640 1 -- 172.21.15.33:0/548281164 >> v1:172.21.15.33:6789/0 conn(0x7f22b8006c90 legacy=0x7f22b80cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.309+0000 7f22c6596640 1 -- 172.21.15.33:0/548281164 shutdown_connections 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.309+0000 7f22c6596640 1 -- 172.21.15.33:0/548281164 >> 172.21.15.33:0/548281164 conn(0x7f22b8005910 msgr2=0x7f22b8005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.309+0000 7f22c6596640 1 -- 172.21.15.33:0/548281164 shutdown_connections 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.309+0000 7f22c6596640 1 -- 172.21.15.33:0/548281164 wait complete. 2024-03-29T11:41:41.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.310+0000 7f22c6596640 10 monclient: build_initial_monmap 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.310+0000 7f22c6596640 10 monclient: monmap: 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:41.313 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.310+0000 7f22c6596640 1 librados: starting msgr at 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.310+0000 7f22c6596640 1 librados: starting objecter 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 1 Processor -- start 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 1 -- start start 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 1 librados: setting wanted keys 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 1 librados: calling monclient init 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 10 monclient: init 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:41.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:41.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.311+0000 7f22c6596640 10 monclient(hunting): picked mon.noname-c con 0x7f22b80068b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:41.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.315+0000 7f22c6596640 10 monclient(hunting): picked mon.noname-a con 0x7f22b802b4e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:41.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.315+0000 7f22c6596640 10 monclient(hunting): picked mon.noname-b con 0x7f22b8152ce0 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:41.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.315+0000 7f22c6596640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b80be8a0 con 0x7f22b802b4e0 2024-03-29T11:41:41.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.315+0000 7f22c6596640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b8160690 con 0x7f22b8152ce0 2024-03-29T11:41:41.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.315+0000 7f22c6596640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b80c6a60 con 0x7f22b80068b0 2024-03-29T11:41:41.317 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.315+0000 7f22c6596640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:41.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.315+0000 7f22c6596640 10 monclient(hunting): authenticate will time out at 1257.579318s 2024-03-29T11:41:41.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.316+0000 7f22bffff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f22b80068b0 0x7f22b80c15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:33101/0 (socket says 172.21.15.33:33101) 2024-03-29T11:41:41.319 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.316+0000 7f22bffff640 1 -- 172.21.15.33:0/94609369 learned_addr learned my addr 172.21.15.33:0/94609369 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 10 client.?.objecter ms_handle_connect 0x7f22b80068b0 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212872557 0 0) 0x7f22b80c6a60 con 0x7f22b80068b0 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:41:41.320 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.317+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a4004eb0 con 0x7f22b80068b0 2024-03-29T11:41:41.321 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.319+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 801328559 0 0) 0x7f22a4004eb0 con 0x7f22b80068b0 2024-03-29T11:41:41.321 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.319+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22b80c6a60 con 0x7f22b80068b0 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.319+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f22c004fdd0 con 0x7f22b80068b0 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.319+0000 7f22bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.319+0000 7f22bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.319+0000 7f22bcff9640 10 monclient(hunting): dump: 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:41.322 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:41.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.319+0000 7f22bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:41.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.321+0000 7f22bcff9640 10 client.?.objecter ms_handle_connect 0x7f22b802b4e0 2024-03-29T11:41:41.323 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.321+0000 7f22bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 client.?.objecter ms_handle_connect 0x7f22b8152ce0 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520697627 0 0) 0x7f22b80be8a0 con 0x7f22b802b4e0 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a4004eb0 con 0x7f22b802b4e0 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006892880 0 0) 0x7f22b8160690 con 0x7f22b8152ce0 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:41.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b80be8a0 con 0x7f22b8152ce0 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1274315038 0 0) 0x7f22b80c6a60 con 0x7f22b80068b0 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 >> v1:172.21.15.153:6789/0 conn(0x7f22b8152ce0 legacy=0x7f22b80be190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.324+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 >> v1:172.21.15.33:6789/0 conn(0x7f22b802b4e0 legacy=0x7f22b80bda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 1 monclient: found mon.c 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b8160b70 con 0x7f22b80068b0 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 10 monclient: _finish_auth 0 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:41:41.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 10 monclient: _renew_subs 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b80c1cf0 con 0x7f22b80068b0 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 10 monclient: _renew_subs 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b80c5f50 con 0x7f22b80068b0 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.325+0000 7f22c6596640 1 librados: init done 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.326+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22c004f400 con 0x7f22b80068b0 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.326+0000 7f22bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.326+0000 7f22be7fc640 4 set_mon_vals no callback set 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.327+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f22c0075710 con 0x7f22b80068b0 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.327+0000 7f22bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.327+0000 7f22bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.327+0000 7f22bcff9640 10 monclient: dump: 2024-03-29T11:41:41.332 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:41.333 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:41.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.329+0000 7f22aa7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:41.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.329+0000 7f22aa7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:41.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.329+0000 7f22aa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:41.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.329+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b8004cf0 con 0x7f22b80068b0 2024-03-29T11:41:41.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.329+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f22c0067bf0 con 0x7f22b80068b0 2024-03-29T11:41:41.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.330+0000 7f22bf7fe640 10 monclient: get_auth_request con 0x7f22a406db90 auth_method 0 2024-03-29T11:41:41.335 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.331+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f22c00ff510 con 0x7f22b80068b0 2024-03-29T11:41:41.335 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.331+0000 7f22bcff9640 10 client.4511.objecter ms_dispatch 0x7f22c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:41.335 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.331+0000 7f22bcff9640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:41.335 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.331+0000 7f22bcff9640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:41.335 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.331+0000 7f22bcff9640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:41:41.335 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.331+0000 7f22bcff9640 10 client.4511.objecter ms_handle_connect 0x7f22a406db90 2024-03-29T11:41:41.343 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.341+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22c00cadf0 con 0x7f22b80068b0 2024-03-29T11:41:41.343 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.341+0000 7f22bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:41.343 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.341+0000 7f22bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:41.676 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.674+0000 7f22aa7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:41.676 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.674+0000 7f22aa7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:41.676 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.674+0000 7f22aa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:41.677 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.674+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22b80c6250 con 0x7f22b80068b0 2024-03-29T11:41:41.680 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.678+0000 7f22bcff9640 1 -- 172.21.15.33:0/94609369 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f22c00ccd20 con 0x7f22b80068b0 2024-03-29T11:41:41.680 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.678+0000 7f22bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:41.680 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.678+0000 7f22bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:41.680 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:41.680 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:41.684 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.681+0000 7f22aa7fc640 10 librados: watch_flush enter 2024-03-29T11:41:41.684 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.681+0000 7f22aa7fc640 10 librados: watch_flush exit 2024-03-29T11:41:41.684 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.681+0000 7f22aa7fc640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:41:41.684 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.681+0000 7f22aa7fc640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:41:41.684 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.682+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 >> v1:172.21.15.153:6816/2737028074 conn(0x7f22a406db90 legacy=0x7f22a4070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.684 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.682+0000 7f22aa7fc640 10 monclient: shutdown 2024-03-29T11:41:41.685 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.682+0000 7f22aa7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:41.685 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.682+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 >> v1:172.21.15.33:6790/0 conn(0x7f22b80068b0 legacy=0x7f22b80c15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:41.685 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.683+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 shutdown_connections 2024-03-29T11:41:41.685 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.683+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 >> 172.21.15.33:0/94609369 conn(0x7f22b8005910 msgr2=0x7f22b8038150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:41.688 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.686+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 shutdown_connections 2024-03-29T11:41:41.688 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.686+0000 7f22aa7fc640 1 -- 172.21.15.33:0/94609369 wait complete. 2024-03-29T11:41:41.688 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:41.686+0000 7f22aa7fc640 1 librados: shutdown 2024-03-29T11:41:41.706 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:48.513 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:48.660 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.658+0000 7f9caac57640 10 monclient: get_monmap_and_config 2024-03-29T11:41:48.660 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.658+0000 7f9caac57640 10 monclient: build_initial_monmap 2024-03-29T11:41:48.660 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.658+0000 7f9caac57640 10 monclient: monmap: 2024-03-29T11:41:48.660 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:48.658698+0000 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:48.658698+0000 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.659+0000 7f9caac57640 1 Processor -- start 2024-03-29T11:41:48.661 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.659+0000 7f9caac57640 1 -- start start 2024-03-29T11:41:48.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.659+0000 7f9caac57640 10 monclient: init 2024-03-29T11:41:48.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.660+0000 7f9caac57640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:48.662 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.660+0000 7f9caac57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:48.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.660+0000 7f9caac57640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca40b0860 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:48.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9caac57640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca413ef30 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:48.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9caac57640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca40af8e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:48.667 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9caac57640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40b0c40 con 0x7f9ca40b0860 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9caac57640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40aed10 con 0x7f9ca413ef30 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9caac57640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40afd50 con 0x7f9ca40af8e0 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9caac57640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9caac57640 10 monclient(hunting): authenticate will time out at 1264.927914s 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9ca89f3640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f9ca40b0860 0x7f9ca413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:41877/0 (socket says 172.21.15.33:41877) 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9ca89f3640 1 -- 172.21.15.33:0/1760428424 learned_addr learned my addr 172.21.15.33:0/1760428424 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.664+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414930610 0 0) 0x7f9ca40b0c40 con 0x7f9ca40b0860 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.665+0000 7f9ca37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.665+0000 7f9ca37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.665+0000 7f9ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.665+0000 7f9ca37fe640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:41:48.668 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.665+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c8c003520 con 0x7f9ca40b0860 2024-03-29T11:41:48.669 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.666+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237304492 0 0) 0x7f9c8c003520 con 0x7f9ca40b0860 2024-03-29T11:41:48.669 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.666+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 >> v1:172.21.15.33:6790/0 conn(0x7f9ca40af8e0 legacy=0x7f9ca4149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 >> v1:172.21.15.153:6789/0 conn(0x7f9ca413ef30 legacy=0x7f9ca40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 1 monclient: found mon.noname-a 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40ab1b0 con 0x7f9ca40b0860 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:48.671 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9c98001690 con 0x7f9ca40b0860 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9caac57640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9caac57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9ca37fe640 10 monclient: dump: 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:48.672 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:48.673 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.668+0000 7f9caac57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.670+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c98001920 con 0x7f9ca40b0860 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.670+0000 7f9ca37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.670+0000 7f9ca37fe640 1 -- 172.21.15.33:0/1760428424 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9c98001cb0 con 0x7f9ca40b0860 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.670+0000 7f9caac57640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.671+0000 7f9caac57640 4 set_mon_vals no callback set 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.672+0000 7f9caac57640 10 monclient: shutdown 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.672+0000 7f9ca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.672+0000 7f9ca37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:48.674 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.672+0000 7f9ca37fe640 10 monclient: dump: 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.672+0000 7f9caac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:48.675 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.672+0000 7f9caac57640 1 -- 172.21.15.33:0/1760428424 >> v1:172.21.15.33:6789/0 conn(0x7f9ca40b0860 legacy=0x7f9ca413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.677 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.674+0000 7f9caac57640 1 -- 172.21.15.33:0/1760428424 shutdown_connections 2024-03-29T11:41:48.677 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.675+0000 7f9caac57640 1 -- 172.21.15.33:0/1760428424 >> 172.21.15.33:0/1760428424 conn(0x7f9ca406b660 msgr2=0x7f9ca406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:48.677 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.675+0000 7f9caac57640 1 -- 172.21.15.33:0/1760428424 shutdown_connections 2024-03-29T11:41:48.678 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.676+0000 7f9caac57640 1 -- 172.21.15.33:0/1760428424 wait complete. 2024-03-29T11:41:48.680 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.677+0000 7f9caac57640 10 monclient: build_initial_monmap 2024-03-29T11:41:48.680 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.677+0000 7f9caac57640 10 monclient: monmap: 2024-03-29T11:41:48.680 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:48.680 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:48.680 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:48.680 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:48.680 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.677+0000 7f9caac57640 1 librados: starting msgr at 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.677+0000 7f9caac57640 1 librados: starting objecter 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.677+0000 7f9caac57640 1 Processor -- start 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.678+0000 7f9caac57640 1 -- start start 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.678+0000 7f9caac57640 1 librados: setting wanted keys 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.678+0000 7f9caac57640 1 librados: calling monclient init 2024-03-29T11:41:48.681 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.678+0000 7f9caac57640 10 monclient: init 2024-03-29T11:41:48.682 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.679+0000 7f9caac57640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:48.682 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.679+0000 7f9caac57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:48.682 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.679+0000 7f9caac57640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca40af8e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:48.685 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.683+0000 7f9caac57640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca40b0860 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:48.685 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.683+0000 7f9ca89f3640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f9ca40af8e0 0x7f9ca4071120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:42119/0 (socket says 172.21.15.33:42119) 2024-03-29T11:41:48.686 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.683+0000 7f9ca89f3640 1 -- 172.21.15.33:0/2542050981 learned_addr learned my addr 172.21.15.33:0/2542050981 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:48.688 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.683+0000 7f9ca17fa640 10 client.?.objecter ms_handle_connect 0x7f9ca40af8e0 2024-03-29T11:41:48.688 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.686+0000 7f9ca17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:48.688 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.686+0000 7f9caac57640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca413ef30 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:48.689 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.686+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca4136db0 con 0x7f9ca40af8e0 2024-03-29T11:41:48.689 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.686+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca4071a10 con 0x7f9ca40b0860 2024-03-29T11:41:48.689 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.686+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca4070a20 con 0x7f9ca413ef30 2024-03-29T11:41:48.689 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.686+0000 7f9caac57640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:48.689 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.686+0000 7f9caac57640 10 monclient(hunting): authenticate will time out at 1264.949870s 2024-03-29T11:41:48.694 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 10 client.?.objecter ms_handle_connect 0x7f9ca40b0860 2024-03-29T11:41:48.694 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 10 client.?.objecter ms_handle_connect 0x7f9ca413ef30 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013521731 0 0) 0x7f9ca4136db0 con 0x7f9ca40af8e0 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.687+0000 7f9ca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c90004eb0 con 0x7f9ca40af8e0 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902154620 0 0) 0x7f9ca4070a20 con 0x7f9ca413ef30 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:41:48.695 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.689+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca4136db0 con 0x7f9ca413ef30 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299502805 0 0) 0x7f9ca4136db0 con 0x7f9ca413ef30 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ca4070a20 con 0x7f9ca413ef30 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9c9c001b10 con 0x7f9ca413ef30 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient(hunting): dump: 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:48.696 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 818946600 0 0) 0x7f9ca4070a20 con 0x7f9ca413ef30 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 >> v1:172.21.15.153:6789/0 conn(0x7f9ca40b0860 legacy=0x7f9ca406f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 >> v1:172.21.15.33:6789/0 conn(0x7f9ca40af8e0 legacy=0x7f9ca4071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 monclient: found mon.c 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca4070d00 con 0x7f9ca413ef30 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9caac57640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:41:48.697 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9caac57640 10 monclient: _renew_subs 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9caac57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca406e910 con 0x7f9ca413ef30 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c9c002000 con 0x7f9ca413ef30 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9c9c003b10 con 0x7f9ca413ef30 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.690+0000 7f9ca17fa640 10 monclient: dump: 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:48.698 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.691+0000 7f9caac57640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.691+0000 7f9caac57640 10 monclient: _renew_subs 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.691+0000 7f9caac57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.691+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca406f7e0 con 0x7f9ca413ef30 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.691+0000 7f9caac57640 1 librados: init done 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.691+0000 7f9ca2ffd640 4 set_mon_vals no callback set 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.692+0000 7f9caac57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.692+0000 7f9caac57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.692+0000 7f9caac57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:48.699 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.692+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca4075f10 con 0x7f9ca413ef30 2024-03-29T11:41:48.700 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.693+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f9c9c019ae0 con 0x7f9ca413ef30 2024-03-29T11:41:48.700 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.693+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f9c9c083a90 con 0x7f9ca413ef30 2024-03-29T11:41:48.701 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.694+0000 7f9ca17fa640 10 client.4514.objecter ms_dispatch 0x7f9ca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:48.701 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.694+0000 7f9ca89f3640 10 monclient: get_auth_request con 0x7f9c9006dbd0 auth_method 0 2024-03-29T11:41:48.701 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.694+0000 7f9ca17fa640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:48.701 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.694+0000 7f9ca17fa640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:48.701 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.694+0000 7f9ca17fa640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:41:48.701 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.694+0000 7f9ca17fa640 10 client.4514.objecter ms_handle_connect 0x7f9c9006dbd0 2024-03-29T11:41:48.707 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.705+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c9c04f370 con 0x7f9ca413ef30 2024-03-29T11:41:48.707 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.705+0000 7f9ca17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:48.707 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:48.705+0000 7f9ca17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:49.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.041+0000 7f9caac57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:49.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.041+0000 7f9caac57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:49.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.041+0000 7f9caac57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:49.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.041+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca4072330 con 0x7f9ca413ef30 2024-03-29T11:41:49.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.042+0000 7f9ca17fa640 1 -- 172.21.15.33:0/2542050981 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f9c9c0512a0 con 0x7f9ca413ef30 2024-03-29T11:41:49.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.042+0000 7f9ca17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:49.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.042+0000 7f9ca17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:49.048 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:49.049 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:49.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 10 librados: watch_flush enter 2024-03-29T11:41:49.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 10 librados: watch_flush exit 2024-03-29T11:41:49.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:41:49.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:41:49.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 >> v1:172.21.15.153:6816/2737028074 conn(0x7f9c9006dbd0 legacy=0x7f9c90070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:49.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 10 monclient: shutdown 2024-03-29T11:41:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 >> v1:172.21.15.33:6790/0 conn(0x7f9ca413ef30 legacy=0x7f9ca41dea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.046+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 shutdown_connections 2024-03-29T11:41:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.047+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 >> 172.21.15.33:0/2542050981 conn(0x7f9ca406b660 msgr2=0x7f9ca413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.047+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 shutdown_connections 2024-03-29T11:41:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.047+0000 7f9caac57640 1 -- 172.21.15.33:0/2542050981 wait complete. 2024-03-29T11:41:49.053 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:49.047+0000 7f9caac57640 1 librados: shutdown 2024-03-29T11:41:49.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:55.872 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.026+0000 7f5721e45640 10 monclient: get_monmap_and_config 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.026+0000 7f5721e45640 10 monclient: build_initial_monmap 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.026+0000 7f5721e45640 10 monclient: monmap: 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:41:56.027348+0000 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:41:56.027348+0000 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:56.029 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:56.030 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:56.030 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:41:56.030 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:41:56.030 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:56.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.027+0000 7f5721e45640 1 Processor -- start 2024-03-29T11:41:56.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.027+0000 7f5721e45640 1 -- start start 2024-03-29T11:41:56.030 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.027+0000 7f5721e45640 10 monclient: init 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.028+0000 7f5721e45640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.028+0000 7f5721e45640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.028+0000 7f5721e45640 10 monclient(hunting): picked mon.noname-b con 0x7f571c0b4790 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.028+0000 7f5721e45640 10 monclient(hunting): picked mon.noname-c con 0x7f571c0aea90 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f5721e45640 10 monclient(hunting): picked mon.noname-a con 0x7f571c0b39f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f5721e45640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0b0c60 con 0x7f571c0b39f0 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f5721e45640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0af020 con 0x7f571c0aea90 2024-03-29T11:41:56.031 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f5721e45640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c13fbd0 con 0x7f571c0b4790 2024-03-29T11:41:56.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f5721e45640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:56.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f5721e45640 10 monclient(hunting): authenticate will time out at 1272.292726s 2024-03-29T11:41:56.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f571b7fe640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f571c0b4790 0x7f571c0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:59452/0 (socket says 172.21.15.33:59452) 2024-03-29T11:41:56.032 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.029+0000 7f571b7fe640 1 -- 172.21.15.33:0/767902847 learned_addr learned my addr 172.21.15.33:0/767902847 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:56.034 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510806769 0 0) 0x7f571c0af020 con 0x7f571c0aea90 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5704003520 con 0x7f571c0aea90 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292221 0 0) 0x7f571c13fbd0 con 0x7f571c0b4790 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c0af020 con 0x7f571c0b4790 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105070716 0 0) 0x7f571c0b0c60 con 0x7f571c0b39f0 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:56.035 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c13fbd0 con 0x7f571c0b39f0 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1495967738 0 0) 0x7f5704003520 con 0x7f571c0aea90 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 >> v1:172.21.15.33:6790/0 conn(0x7f571c0b4790 legacy=0x7f571c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 >> v1:172.21.15.33:6789/0 conn(0x7f571c0b39f0 legacy=0x7f571c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c13fd60 con 0x7f571c0aea90 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:56.036 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5710001690 con 0x7f571c0aea90 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f571a7fc640 10 monclient: dump: 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f5721e45640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.030+0000 7f5721e45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:56.037 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.031+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57100017f0 con 0x7f571c0aea90 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.031+0000 7f571a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.031+0000 7f571a7fc640 1 -- 172.21.15.33:0/767902847 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5710001c00 con 0x7f571c0aea90 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.032+0000 7f571a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.032+0000 7f571a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.032+0000 7f571a7fc640 10 monclient: dump: 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:56.038 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:56.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.032+0000 7f5721e45640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:56.039 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.032+0000 7f5721e45640 4 set_mon_vals no callback set 2024-03-29T11:41:56.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.033+0000 7f5721e45640 10 monclient: shutdown 2024-03-29T11:41:56.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.033+0000 7f5721e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:56.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.033+0000 7f5721e45640 1 -- 172.21.15.33:0/767902847 >> v1:172.21.15.153:6789/0 conn(0x7f571c0aea90 legacy=0x7f571c0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.033+0000 7f5721e45640 1 -- 172.21.15.33:0/767902847 shutdown_connections 2024-03-29T11:41:56.040 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.033+0000 7f5721e45640 1 -- 172.21.15.33:0/767902847 >> 172.21.15.33:0/767902847 conn(0x7f571c066a00 msgr2=0x7f571c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.033+0000 7f5721e45640 1 -- 172.21.15.33:0/767902847 shutdown_connections 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.033+0000 7f5721e45640 1 -- 172.21.15.33:0/767902847 wait complete. 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.035+0000 7f5721e45640 10 monclient: build_initial_monmap 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.035+0000 7f5721e45640 10 monclient: monmap: 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:56.041 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.035+0000 7f5721e45640 1 librados: starting msgr at 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.035+0000 7f5721e45640 1 librados: starting objecter 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.035+0000 7f5721e45640 1 Processor -- start 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.036+0000 7f5721e45640 1 -- start start 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.036+0000 7f5721e45640 1 librados: setting wanted keys 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.036+0000 7f5721e45640 1 librados: calling monclient init 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.036+0000 7f5721e45640 10 monclient: init 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.036+0000 7f5721e45640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.036+0000 7f5721e45640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 10 monclient(hunting): picked mon.noname-a con 0x7f571c0b39f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 10 monclient(hunting): picked mon.noname-c con 0x7f571c0b4790 addr v1:172.21.15.33:6790/0 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 10 monclient(hunting): picked mon.noname-b con 0x7f571c133f80 addr v1:172.21.15.153:6789/0 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f571b7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f571c0b39f0 0x7f571c1dac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:38230/0 (socket says 172.21.15.33:38230) 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f571b7fe640 1 -- 172.21.15.33:0/3581485183 learned_addr learned my addr 172.21.15.33:0/3581485183 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:41:56.042 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c142e80 con 0x7f571c0b39f0 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c1db350 con 0x7f571c133f80 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5703fff640 10 client.?.objecter ms_handle_connect 0x7f571c0b39f0 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5703fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c1db4e0 con 0x7f571c0b4790 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5721e45640 10 monclient(hunting): authenticate will time out at 1272.301045s 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5703fff640 10 client.?.objecter ms_handle_connect 0x7f571c0b4790 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5703fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5703fff640 10 client.?.objecter ms_handle_connect 0x7f571c133f80 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.037+0000 7f5703fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455487164 0 0) 0x7f571c1db4e0 con 0x7f571c0b4790 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5708004eb0 con 0x7f571c0b4790 2024-03-29T11:41:56.043 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744370158 0 0) 0x7f571c1db350 con 0x7f571c133f80 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c1db4e0 con 0x7f571c133f80 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680573109 0 0) 0x7f571c142e80 con 0x7f571c0b39f0 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:41:56.044 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c1db350 con 0x7f571c0b39f0 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832213002 0 0) 0x7f5708004eb0 con 0x7f571c0b4790 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f571c142e80 con 0x7f571c0b4790 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 330762121 0 0) 0x7f571c1db4e0 con 0x7f571c133f80 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5708004eb0 con 0x7f571c133f80 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5710001c00 con 0x7f571c0b4790 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): dump: 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.046 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5714001b60 con 0x7f571c133f80 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.038+0000 7f5703fff640 10 monclient(hunting): dump: 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.047 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1836920112 0 0) 0x7f571c142e80 con 0x7f571c0b4790 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 >> v1:172.21.15.153:6789/0 conn(0x7f571c133f80 legacy=0x7f571c1d48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 >> v1:172.21.15.33:6789/0 conn(0x7f571c0b39f0 legacy=0x7f571c1dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 1 monclient: found mon.c 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c1e4dc0 con 0x7f571c0b4790 2024-03-29T11:41:56.048 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 10 monclient: _finish_auth 0 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 10 monclient: _renew_subs 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c1e2da0 con 0x7f571c0b4790 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 10 monclient: _renew_subs 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c1e32d0 con 0x7f571c0b4790 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.039+0000 7f5721e45640 1 librados: init done 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5710001050 con 0x7f571c0b4790 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5703fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5710003a40 con 0x7f571c0b4790 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:56.049 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5703fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5703fff640 10 monclient: dump: 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5719ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.040+0000 7f5721e45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:56.050 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.042+0000 7f5721e45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:56.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.042+0000 7f5721e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:56.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.042+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56fc005080 con 0x7f571c0b4790 2024-03-29T11:41:56.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.043+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f5710019a10 con 0x7f571c0b4790 2024-03-29T11:41:56.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.044+0000 7f571b7fe640 10 monclient: get_auth_request con 0x7f570806dfc0 auth_method 0 2024-03-29T11:41:56.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.045+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f5710084940 con 0x7f571c0b4790 2024-03-29T11:41:56.051 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.045+0000 7f5703fff640 10 client.4520.objecter ms_dispatch 0x7f571c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:56.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.045+0000 7f5703fff640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:56.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.045+0000 7f5703fff640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:56.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.045+0000 7f5703fff640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:41:56.052 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.045+0000 7f5703fff640 10 client.4520.objecter ms_handle_connect 0x7f570806dfc0 2024-03-29T11:41:56.062 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.060+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f571004f1f0 con 0x7f571c0b4790 2024-03-29T11:41:56.062 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.060+0000 7f5703fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:56.062 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.060+0000 7f5703fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:56.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.488+0000 7f5721e45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:56.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.488+0000 7f5721e45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:56.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.488+0000 7f5721e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:41:56.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.488+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56fc005330 con 0x7f571c0b4790 2024-03-29T11:41:56.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.490+0000 7f5703fff640 1 -- 172.21.15.33:0/3581485183 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f5710051120 con 0x7f571c0b4790 2024-03-29T11:41:56.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.490+0000 7f5703fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:56.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.490+0000 7f5703fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:56.493 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:41:56.493 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 10 librados: watch_flush enter 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 10 librados: watch_flush exit 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 >> v1:172.21.15.153:6816/2737028074 conn(0x7f570806dfc0 legacy=0x7f5708070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 10 monclient: shutdown 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:56.497 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 >> v1:172.21.15.33:6790/0 conn(0x7f571c0b4790 legacy=0x7f571c1d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:56.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 shutdown_connections 2024-03-29T11:41:56.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 >> 172.21.15.33:0/3581485183 conn(0x7f571c066a00 msgr2=0x7f571c0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:56.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 shutdown_connections 2024-03-29T11:41:56.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 1 -- 172.21.15.33:0/3581485183 wait complete. 2024-03-29T11:41:56.498 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:41:56.495+0000 7f5721e45640 1 librados: shutdown 2024-03-29T11:41:56.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:03.320 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:03.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.469+0000 7fc6e4c5c640 10 monclient: get_monmap_and_config 2024-03-29T11:42:03.472 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.469+0000 7fc6e4c5c640 10 monclient: build_initial_monmap 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.469+0000 7fc6e4c5c640 10 monclient: monmap: 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:03.469992+0000 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:03.469992+0000 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:03.473 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:03.474 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.471+0000 7fc6e4c5c640 1 Processor -- start 2024-03-29T11:42:03.475 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.473+0000 7fc6e4c5c640 1 -- start start 2024-03-29T11:42:03.475 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.473+0000 7fc6e4c5c640 10 monclient: init 2024-03-29T11:42:03.476 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.474+0000 7fc6e4c5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:03.476 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.474+0000 7fc6e4c5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:03.476 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.474+0000 7fc6e4c5c640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e00ae250 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:03.477 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.475+0000 7fc6e4c5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e013f970 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:03.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.476+0000 7fc6e4c5c640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e0142800 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:03.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.476+0000 7fc6e4c5c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e00afe00 con 0x7fc6e0142800 2024-03-29T11:42:03.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.476+0000 7fc6e4c5c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e0142c10 con 0x7fc6e00ae250 2024-03-29T11:42:03.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.477+0000 7fc6e4c5c640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e00afb20 con 0x7fc6e013f970 2024-03-29T11:42:03.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.477+0000 7fc6e4c5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:03.479 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.477+0000 7fc6e4c5c640 10 monclient(hunting): authenticate will time out at 1279.740691s 2024-03-29T11:42:03.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.480+0000 7fc6de59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc6e00ae250 0x7fc6e013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:46929/0 (socket says 172.21.15.33:46929) 2024-03-29T11:42:03.482 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.480+0000 7fc6de59c640 1 -- 172.21.15.33:0/3193367430 learned_addr learned my addr 172.21.15.33:0/3193367430 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:03.486 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230615909 0 0) 0x7fc6e00afb20 con 0x7fc6e013f970 2024-03-29T11:42:03.486 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c8003520 con 0x7fc6e013f970 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862376933 0 0) 0x7fc6e00afe00 con 0x7fc6e0142800 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e00afb20 con 0x7fc6e0142800 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461574077 0 0) 0x7fc6e0142c10 con 0x7fc6e00ae250 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:42:03.487 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e00afe00 con 0x7fc6e00ae250 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392456216 0 0) 0x7fc6e00afb20 con 0x7fc6e0142800 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 >> v1:172.21.15.33:6790/0 conn(0x7fc6e013f970 legacy=0x7fc6e01420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 >> v1:172.21.15.153:6789/0 conn(0x7fc6e00ae250 legacy=0x7fc6e013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 monclient: found mon.noname-a 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e00aab80 con 0x7fc6e0142800 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient: _finish_auth 0 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fc6d8001970 con 0x7fc6e0142800 2024-03-29T11:42:03.489 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.484+0000 7fc6dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 10 monclient: dump: 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6e4c5c640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:42:03.490 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6e4c5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6d8001b90 con 0x7fc6e0142800 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 1 -- 172.21.15.33:0/3193367430 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fc6d8001f20 con 0x7fc6e0142800 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6dd59a640 10 monclient: dump: 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:03.491 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6e4c5c640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.485+0000 7fc6e4c5c640 4 set_mon_vals no callback set 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.486+0000 7fc6e4c5c640 10 monclient: shutdown 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.486+0000 7fc6e4c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.486+0000 7fc6e4c5c640 1 -- 172.21.15.33:0/3193367430 >> v1:172.21.15.33:6789/0 conn(0x7fc6e0142800 legacy=0x7fc6e014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.486+0000 7fc6e4c5c640 1 -- 172.21.15.33:0/3193367430 shutdown_connections 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.487+0000 7fc6e4c5c640 1 -- 172.21.15.33:0/3193367430 >> 172.21.15.33:0/3193367430 conn(0x7fc6e00b35e0 msgr2=0x7fc6e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:03.492 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.489+0000 7fc6e4c5c640 1 -- 172.21.15.33:0/3193367430 shutdown_connections 2024-03-29T11:42:03.493 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.489+0000 7fc6e4c5c640 1 -- 172.21.15.33:0/3193367430 wait complete. 2024-03-29T11:42:03.493 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.490+0000 7fc6e4c5c640 10 monclient: build_initial_monmap 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.490+0000 7fc6e4c5c640 10 monclient: monmap: 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:03.494 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.491+0000 7fc6e4c5c640 1 librados: starting msgr at 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.491+0000 7fc6e4c5c640 1 librados: starting objecter 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.491+0000 7fc6e4c5c640 1 Processor -- start 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.492+0000 7fc6e4c5c640 1 -- start start 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.492+0000 7fc6e4c5c640 1 librados: setting wanted keys 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.492+0000 7fc6e4c5c640 1 librados: calling monclient init 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.492+0000 7fc6e4c5c640 10 monclient: init 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.493+0000 7fc6e4c5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:03.495 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.493+0000 7fc6e4c5c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:03.496 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.493+0000 7fc6e4c5c640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e00ae250 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:03.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.494+0000 7fc6e4c5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e013f970 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:03.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.494+0000 7fc6e4c5c640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e0142800 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:03.499 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.494+0000 7fc6e4c5c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e01d7ba0 con 0x7fc6e00ae250 2024-03-29T11:42:03.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.494+0000 7fc6e4c5c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e01d78d0 con 0x7fc6e013f970 2024-03-29T11:42:03.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.494+0000 7fc6e4c5c640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e0145b10 con 0x7fc6e0142800 2024-03-29T11:42:03.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.494+0000 7fc6e4c5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:03.500 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.494+0000 7fc6e4c5c640 10 monclient(hunting): authenticate will time out at 1279.758653s 2024-03-29T11:42:03.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ded9d640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fc6e0142800 0x7fc6e013ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:49575/0 (socket says 172.21.15.33:49575) 2024-03-29T11:42:03.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ded9d640 1 -- 172.21.15.33:0/2836964578 learned_addr learned my addr 172.21.15.33:0/2836964578 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:03.501 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 10 client.?.objecter ms_handle_connect 0x7fc6e0142800 2024-03-29T11:42:03.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:03.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381550540 0 0) 0x7fc6e0145b10 con 0x7fc6e0142800 2024-03-29T11:42:03.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:03.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:03.502 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:03.503 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:42:03.503 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.499+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c0004ee0 con 0x7fc6e0142800 2024-03-29T11:42:03.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.500+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 201730765 0 0) 0x7fc6c0004ee0 con 0x7fc6e0142800 2024-03-29T11:42:03.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.500+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e0145b10 con 0x7fc6e0142800 2024-03-29T11:42:03.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.500+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fc6d8001f60 con 0x7fc6e0142800 2024-03-29T11:42:03.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.500+0000 7fc6ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.504 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.500+0000 7fc6ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:03.505 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.500+0000 7fc6ceffd640 10 monclient(hunting): dump: 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.500+0000 7fc6ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1876523115 0 0) 0x7fc6e0145b10 con 0x7fc6e0142800 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 >> v1:172.21.15.153:6789/0 conn(0x7fc6e013f970 legacy=0x7fc6e013a740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 >> v1:172.21.15.33:6789/0 conn(0x7fc6e00ae250 legacy=0x7fc6e01d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.506 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 1 monclient: found mon.c 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e0145ca0 con 0x7fc6e0142800 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.501+0000 7fc6ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.502+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6d8003c20 con 0x7fc6e0142800 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.502+0000 7fc6ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.502+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fc6d8004030 con 0x7fc6e0142800 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.502+0000 7fc6ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.502+0000 7fc6ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.502+0000 7fc6ceffd640 10 monclient: dump: 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:03.507 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:03.508 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.502+0000 7fc6dcd99640 4 set_mon_vals no callback set 2024-03-29T11:42:03.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.505+0000 7fc6e4c5c640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:42:03.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.505+0000 7fc6e4c5c640 10 monclient: _renew_subs 2024-03-29T11:42:03.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.505+0000 7fc6e4c5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:03.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.505+0000 7fc6e4c5c640 1 -- 172.21.15.33:0/2836964578 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e01e29e0 con 0x7fc6e0142800 2024-03-29T11:42:03.509 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.506+0000 7fc6e4c5c640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.506+0000 7fc6e4c5c640 10 monclient: _renew_subs 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.506+0000 7fc6e4c5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.506+0000 7fc6e4c5c640 1 -- 172.21.15.33:0/2836964578 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e01e2d50 con 0x7fc6e0142800 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.506+0000 7fc6e4c5c640 1 librados: init done 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.507+0000 7fc6ccff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.507+0000 7fc6ccff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.507+0000 7fc6ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:03.510 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.507+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b8005080 con 0x7fc6e0142800 2024-03-29T11:42:03.511 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.509+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fc6d8004190 con 0x7fc6e0142800 2024-03-29T11:42:03.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.512+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fc6d8084fd0 con 0x7fc6e0142800 2024-03-29T11:42:03.518 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.512+0000 7fc6ceffd640 10 client.4526.objecter ms_dispatch 0x7fc6e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:03.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.512+0000 7fc6ceffd640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:03.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.512+0000 7fc6ceffd640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:03.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.512+0000 7fc6ceffd640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:42:03.519 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.513+0000 7fc6de59c640 10 monclient: get_auth_request con 0x7fc6c006dd10 auth_method 0 2024-03-29T11:42:03.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.518+0000 7fc6ceffd640 10 client.4526.objecter ms_handle_connect 0x7fc6c006dd10 2024-03-29T11:42:03.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.518+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6d80508b0 con 0x7fc6e0142800 2024-03-29T11:42:03.520 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.518+0000 7fc6ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:03.521 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.518+0000 7fc6ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:03.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.912+0000 7fc6ccff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:03.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.912+0000 7fc6ccff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:03.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.912+0000 7fc6ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:03.915 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.912+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6b8005330 con 0x7fc6e0142800 2024-03-29T11:42:03.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.913+0000 7fc6ceffd640 1 -- 172.21.15.33:0/2836964578 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fc6d80527e0 con 0x7fc6e0142800 2024-03-29T11:42:03.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.913+0000 7fc6ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:03.916 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.913+0000 7fc6ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:03.916 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:03.916 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 10 librados: watch_flush enter 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 10 librados: watch_flush exit 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 >> v1:172.21.15.153:6816/2737028074 conn(0x7fc6c006dd10 legacy=0x7fc6c00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 10 monclient: shutdown 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:03.920 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.917+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 >> v1:172.21.15.33:6790/0 conn(0x7fc6e0142800 legacy=0x7fc6e013ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:03.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.918+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 shutdown_connections 2024-03-29T11:42:03.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.918+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 >> 172.21.15.33:0/2836964578 conn(0x7fc6e00b35e0 msgr2=0x7fc6e00b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:03.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.918+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 shutdown_connections 2024-03-29T11:42:03.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.918+0000 7fc6ccff9640 1 -- 172.21.15.33:0/2836964578 wait complete. 2024-03-29T11:42:03.921 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:03.918+0000 7fc6ccff9640 1 librados: shutdown 2024-03-29T11:42:03.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:10.744 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:10.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.890+0000 7f186a04d640 10 monclient: get_monmap_and_config 2024-03-29T11:42:10.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.890+0000 7f186a04d640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.891+0000 7f186a04d640 10 monclient: monmap: 2024-03-29T11:42:10.894 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:10.894 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:10.892607+0000 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:10.892607+0000 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:10.895 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:10.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.894+0000 7f186a04d640 1 Processor -- start 2024-03-29T11:42:10.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.894+0000 7f186a04d640 1 -- start start 2024-03-29T11:42:10.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.894+0000 7f186a04d640 10 monclient: init 2024-03-29T11:42:10.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.895+0000 7f186a04d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:10.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 10 monclient(hunting): picked mon.noname-c con 0x7f18640b2520 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:10.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 10 monclient(hunting): picked mon.noname-b con 0x7f18640b2900 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:10.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 10 monclient(hunting): picked mon.noname-a con 0x7f18640a9c70 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:10.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864142170 con 0x7f18640a9c70 2024-03-29T11:42:10.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18640b2260 con 0x7f18640b2520 2024-03-29T11:42:10.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18640aaf60 con 0x7f18640b2900 2024-03-29T11:42:10.900 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.896+0000 7f186a04d640 10 monclient(hunting): authenticate will time out at 1287.161280s 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f1863fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f18640a9c70 0x7f186414d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:36060/0 (socket says 172.21.15.33:36060) 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f1863fff640 1 -- 172.21.15.33:0/4197009281 learned_addr learned my addr 172.21.15.33:0/4197009281 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198515780 0 0) 0x7f1864142170 con 0x7f18640a9c70 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f18627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f18627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f18627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f18627fc640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:42:10.901 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.897+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1854003520 con 0x7f18640a9c70 2024-03-29T11:42:10.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3263493678 0 0) 0x7f1854003520 con 0x7f18640a9c70 2024-03-29T11:42:10.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 >> v1:172.21.15.33:6790/0 conn(0x7f18640b2900 legacy=0x7f1864142600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:10.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 >> v1:172.21.15.153:6789/0 conn(0x7f18640b2520 legacy=0x7f18640b1b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:42:10.904 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 1 monclient: found mon.noname-a 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864142d10 con 0x7f18640a9c70 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f185c001a20 con 0x7f18640a9c70 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f18627fc640 10 monclient: dump: 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:10.905 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f186a04d640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:42:10.906 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.901+0000 7f186a04d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.902+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f185c001bf0 con 0x7f18640a9c70 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.902+0000 7f18627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.902+0000 7f186a04d640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.902+0000 7f186a04d640 4 set_mon_vals no callback set 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.903+0000 7f186a04d640 10 monclient: shutdown 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.903+0000 7f186a04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.903+0000 7f186a04d640 1 -- 172.21.15.33:0/4197009281 >> v1:172.21.15.33:6789/0 conn(0x7f18640a9c70 legacy=0x7f186414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.903+0000 7f18627fc640 1 -- 172.21.15.33:0/4197009281 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f185c002040 con 0x7f18640a9c70 2024-03-29T11:42:10.907 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.903+0000 7f18627fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:42:10.908 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.904+0000 7f186a04d640 1 -- 172.21.15.33:0/4197009281 shutdown_connections 2024-03-29T11:42:10.908 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.904+0000 7f186a04d640 1 -- 172.21.15.33:0/4197009281 >> 172.21.15.33:0/4197009281 conn(0x7f186406d180 msgr2=0x7f1864066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.908 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.904+0000 7f186a04d640 1 -- 172.21.15.33:0/4197009281 shutdown_connections 2024-03-29T11:42:10.908 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.904+0000 7f186a04d640 1 -- 172.21.15.33:0/4197009281 wait complete. 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.905+0000 7f186a04d640 10 monclient: build_initial_monmap 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.906+0000 7f186a04d640 10 monclient: monmap: 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:10.909 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:10.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.906+0000 7f186a04d640 1 librados: starting msgr at 2024-03-29T11:42:10.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.906+0000 7f186a04d640 1 librados: starting objecter 2024-03-29T11:42:10.910 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.906+0000 7f186a04d640 1 Processor -- start 2024-03-29T11:42:10.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.907+0000 7f186a04d640 1 -- start start 2024-03-29T11:42:10.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.907+0000 7f186a04d640 1 librados: setting wanted keys 2024-03-29T11:42:10.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.907+0000 7f186a04d640 1 librados: calling monclient init 2024-03-29T11:42:10.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.907+0000 7f186a04d640 10 monclient: init 2024-03-29T11:42:10.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.908+0000 7f186a04d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:10.911 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.908+0000 7f186a04d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:10.913 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.908+0000 7f186a04d640 10 monclient(hunting): picked mon.noname-b con 0x7f18640a9c70 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:10.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.910+0000 7f186a04d640 10 monclient(hunting): picked mon.noname-c con 0x7f18640b2520 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:10.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.914+0000 7f1862ffd640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f18640b2520 0x7f186412ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:49141/0 (socket says 172.21.15.33:49141) 2024-03-29T11:42:10.917 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.914+0000 7f1862ffd640 1 -- 172.21.15.33:0/2057903267 learned_addr learned my addr 172.21.15.33:0/2057903267 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.914+0000 7f186a04d640 10 monclient(hunting): picked mon.noname-a con 0x7f186413c520 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.914+0000 7f184ffff640 10 client.?.objecter ms_handle_connect 0x7f18640b2520 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.914+0000 7f184ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864137ec0 con 0x7f186413c520 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 10 client.?.objecter ms_handle_connect 0x7f186413c520 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864138200 con 0x7f18640a9c70 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186413f3c0 con 0x7f18640b2520 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f186a04d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f186a04d640 10 monclient(hunting): authenticate will time out at 1287.180173s 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671483647 0 0) 0x7f1864137ec0 con 0x7f186413c520 2024-03-29T11:42:10.923 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.915+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848004eb0 con 0x7f186413c520 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074915562 0 0) 0x7f186413f3c0 con 0x7f18640b2520 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1864137ec0 con 0x7f18640b2520 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108531115 0 0) 0x7f1848004eb0 con 0x7f186413c520 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f186413f3c0 con 0x7f186413c520 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f185c001f00 con 0x7f186413c520 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:10.924 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2204679613 0 0) 0x7f186413f3c0 con 0x7f186413c520 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 >> v1:172.21.15.33:6790/0 conn(0x7f18640b2520 legacy=0x7f186412ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 >> v1:172.21.15.153:6789/0 conn(0x7f18640a9c70 legacy=0x7f186413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.925 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 monclient: found mon.a 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186413e750 con 0x7f186413c520 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f184ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f186a04d640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.916+0000 7f186a04d640 10 monclient: _renew_subs 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f186a04d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1864145910 con 0x7f186413c520 2024-03-29T11:42:10.926 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f186a04d640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f186a04d640 10 monclient: _renew_subs 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f186a04d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f186412f4a0 con 0x7f186413c520 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f186a04d640 1 librados: init done 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f185c002060 con 0x7f186413c520 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f184ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f185c005140 con 0x7f186413c520 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f1861ffb640 4 set_mon_vals no callback set 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.917+0000 7f184ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.918+0000 7f184ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.918+0000 7f184ffff640 10 monclient: dump: 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:10.927 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:10.928 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:10.928 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:10.928 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:10.928 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:10.928 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:10.928 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.919+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f185c01b110 con 0x7f186413c520 2024-03-29T11:42:10.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.919+0000 7f186a04d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.919+0000 7f186a04d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.919+0000 7f186a04d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:10.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.919+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1844005080 con 0x7f186413c520 2024-03-29T11:42:10.929 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.920+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f185c084ec0 con 0x7f186413c520 2024-03-29T11:42:10.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.920+0000 7f184ffff640 10 client.4485.objecter ms_dispatch 0x7f1864000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:10.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.920+0000 7f184ffff640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:10.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.920+0000 7f184ffff640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:10.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.920+0000 7f184ffff640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-29T11:42:10.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.920+0000 7f18637fe640 10 monclient: get_auth_request con 0x7f184806db20 auth_method 0 2024-03-29T11:42:10.930 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.921+0000 7f184ffff640 10 client.4485.objecter ms_handle_connect 0x7f184806db20 2024-03-29T11:42:10.931 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.928+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f185c0507a0 con 0x7f186413c520 2024-03-29T11:42:10.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.928+0000 7f184ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:10.932 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:10.928+0000 7f184ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:11.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.263+0000 7f186a04d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:11.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.263+0000 7f186a04d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:11.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.263+0000 7f186a04d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:11.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.263+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1844005330 con 0x7f186413c520 2024-03-29T11:42:11.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.265+0000 7f184ffff640 1 -- 172.21.15.33:0/2057903267 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f185c0526d0 con 0x7f186413c520 2024-03-29T11:42:11.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.265+0000 7f184ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:11.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.265+0000 7f184ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:11.268 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:11.269 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:11.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.268+0000 7f186a04d640 10 librados: watch_flush enter 2024-03-29T11:42:11.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.269+0000 7f186a04d640 10 librados: watch_flush exit 2024-03-29T11:42:11.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.269+0000 7f186a04d640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-29T11:42:11.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.269+0000 7f186a04d640 10 client.4485.objecter successfully canceled tick 2024-03-29T11:42:11.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.269+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 >> v1:172.21.15.153:6816/2737028074 conn(0x7f184806db20 legacy=0x7f184806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.269+0000 7f186a04d640 10 monclient: shutdown 2024-03-29T11:42:11.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.269+0000 7f186a04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:11.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.269+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 >> v1:172.21.15.33:6789/0 conn(0x7f186413c520 legacy=0x7f18641e6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:11.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.270+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 shutdown_connections 2024-03-29T11:42:11.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.270+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 >> 172.21.15.33:0/2057903267 conn(0x7f186406d180 msgr2=0x7f18640b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:11.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.270+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 shutdown_connections 2024-03-29T11:42:11.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.270+0000 7f186a04d640 1 -- 172.21.15.33:0/2057903267 wait complete. 2024-03-29T11:42:11.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:11.270+0000 7f186a04d640 1 librados: shutdown 2024-03-29T11:42:11.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:18.097 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:18.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.240+0000 7fe639ab3640 10 monclient: get_monmap_and_config 2024-03-29T11:42:18.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.240+0000 7fe639ab3640 10 monclient: build_initial_monmap 2024-03-29T11:42:18.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.240+0000 7fe639ab3640 10 monclient: monmap: 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:18.242538+0000 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:18.242538+0000 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:18.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.242+0000 7fe639ab3640 1 Processor -- start 2024-03-29T11:42:18.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.243+0000 7fe639ab3640 1 -- start start 2024-03-29T11:42:18.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.243+0000 7fe639ab3640 10 monclient: init 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.244+0000 7fe639ab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.244+0000 7fe639ab3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.244+0000 7fe639ab3640 10 monclient(hunting): picked mon.noname-b con 0x7fe6340a9ac0 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.244+0000 7fe639ab3640 10 monclient(hunting): picked mon.noname-c con 0x7fe63413ee90 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe6337fe640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fe6340a9ac0 0x7fe6340adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:35814/0 (socket says 172.21.15.33:35814) 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe639ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fe6340af900 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe6337fe640 1 -- 172.21.15.33:0/2837054786 learned_addr learned my addr 172.21.15.33:0/2837054786 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe639ab3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6340afd10 con 0x7fe6340af900 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe639ab3640 1 -- 172.21.15.33:0/2837054786 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6340af6f0 con 0x7fe63413ee90 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe639ab3640 1 -- 172.21.15.33:0/2837054786 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6340aac80 con 0x7fe6340a9ac0 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe639ab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:18.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.246+0000 7fe639ab3640 10 monclient(hunting): authenticate will time out at 1294.511450s 2024-03-29T11:42:18.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.249+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365888217 0 0) 0x7fe6340aac80 con 0x7fe6340a9ac0 2024-03-29T11:42:18.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.249+0000 7fe6327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:18.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.249+0000 7fe6327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:18.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.249+0000 7fe6327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:18.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.249+0000 7fe6327fc640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:42:18.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.249+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61c003520 con 0x7fe6340a9ac0 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.250+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509718224 0 0) 0x7fe6340afd10 con 0x7fe6340af900 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6340aac80 con 0x7fe6340af900 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184631536 0 0) 0x7fe6340af6f0 con 0x7fe63413ee90 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:42:18.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6340afd10 con 0x7fe63413ee90 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273646540 0 0) 0x7fe6340aac80 con 0x7fe6340af900 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 >> v1:172.21.15.33:6790/0 conn(0x7fe6340a9ac0 legacy=0x7fe6340adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 >> v1:172.21.15.153:6789/0 conn(0x7fe63413ee90 legacy=0x7fe63413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 monclient: found mon.noname-a 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6340aaf60 con 0x7fe6340af900 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:18.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe62c0019f0 con 0x7fe6340af900 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: dump: 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.259 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe639ab3640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe639ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe62c001b50 con 0x7fe6340af900 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 1 -- 172.21.15.33:0/2837054786 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe62c001fa0 con 0x7fe6340af900 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:18.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe6327fc640 10 monclient: dump: 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe639ab3640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.251+0000 7fe639ab3640 4 set_mon_vals no callback set 2024-03-29T11:42:18.261 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.252+0000 7fe639ab3640 10 monclient: shutdown 2024-03-29T11:42:18.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.252+0000 7fe639ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:18.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.252+0000 7fe639ab3640 1 -- 172.21.15.33:0/2837054786 >> v1:172.21.15.33:6789/0 conn(0x7fe6340af900 legacy=0x7fe634149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.252+0000 7fe639ab3640 1 -- 172.21.15.33:0/2837054786 shutdown_connections 2024-03-29T11:42:18.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.252+0000 7fe639ab3640 1 -- 172.21.15.33:0/2837054786 >> 172.21.15.33:0/2837054786 conn(0x7fe6340b35e0 msgr2=0x7fe6340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:18.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.253+0000 7fe639ab3640 1 -- 172.21.15.33:0/2837054786 shutdown_connections 2024-03-29T11:42:18.262 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.253+0000 7fe639ab3640 1 -- 172.21.15.33:0/2837054786 wait complete. 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.254+0000 7fe639ab3640 10 monclient: build_initial_monmap 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.255+0000 7fe639ab3640 10 monclient: monmap: 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:18.265 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.255+0000 7fe639ab3640 1 librados: starting msgr at 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.255+0000 7fe639ab3640 1 librados: starting objecter 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.255+0000 7fe639ab3640 1 Processor -- start 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.256+0000 7fe639ab3640 1 -- start start 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.256+0000 7fe639ab3640 1 librados: setting wanted keys 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.256+0000 7fe639ab3640 1 librados: calling monclient init 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.256+0000 7fe639ab3640 10 monclient: init 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.257+0000 7fe639ab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.257+0000 7fe639ab3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.257+0000 7fe639ab3640 10 monclient(hunting): picked mon.noname-b con 0x7fe6340a9ac0 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe639ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fe6340af900 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:18.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe639ab3640 10 monclient(hunting): picked mon.noname-c con 0x7fe63413ee90 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe639ab3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6341c9660 con 0x7fe6340af900 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe639ab3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6341d3380 con 0x7fe6340a9ac0 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe639ab3640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6341c9140 con 0x7fe63413ee90 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe639ab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe639ab3640 10 monclient(hunting): authenticate will time out at 1294.523011s 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe633fff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fe63413ee90 0x7fe6341348a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:44010/0 (socket says 172.21.15.33:44010) 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.258+0000 7fe633fff640 1 -- 172.21.15.33:0/272427872 learned_addr learned my addr 172.21.15.33:0/272427872 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 10 client.?.objecter ms_handle_connect 0x7fe63413ee90 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961229813 0 0) 0x7fe6341c9140 con 0x7fe63413ee90 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe620004eb0 con 0x7fe63413ee90 2024-03-29T11:42:18.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387512017 0 0) 0x7fe620004eb0 con 0x7fe63413ee90 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6341c9140 con 0x7fe63413ee90 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe62c0019d0 con 0x7fe63413ee90 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.259+0000 7fe61bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.260+0000 7fe61bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.260+0000 7fe61bfff640 10 monclient(hunting): dump: 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:18.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.260+0000 7fe61bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:18.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.260+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 181096326 0 0) 0x7fe6341c9140 con 0x7fe63413ee90 2024-03-29T11:42:18.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.260+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 >> v1:172.21.15.153:6789/0 conn(0x7fe6340a9ac0 legacy=0x7fe6341c89d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 >> v1:172.21.15.33:6789/0 conn(0x7fe6340af900 legacy=0x7fe6341d2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 1 monclient: found mon.c 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6341c92d0 con 0x7fe63413ee90 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 10 monclient: _finish_auth 0 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe62c003c90 con 0x7fe63413ee90 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.263+0000 7fe61bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fe62c0040e0 con 0x7fe63413ee90 2024-03-29T11:42:18.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe61bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe61bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe61bfff640 10 monclient: dump: 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe6317fa640 4 set_mon_vals no callback set 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 10 monclient: _renew_subs 2024-03-29T11:42:18.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6340b2190 con 0x7fe63413ee90 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 10 monclient: _renew_subs 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe634135190 con 0x7fe63413ee90 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.264+0000 7fe639ab3640 1 librados: init done 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.266+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fe62c0042c0 con 0x7fe63413ee90 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fe62c085090 con 0x7fe63413ee90 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe61bfff640 10 client.4535.objecter ms_dispatch 0x7fe634000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe61bfff640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe61bfff640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe61bfff640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe639ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:18.274 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe639ab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:18.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe639ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:18.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.267+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63413bcd0 con 0x7fe63413ee90 2024-03-29T11:42:18.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.269+0000 7fe6337fe640 10 monclient: get_auth_request con 0x7fe62006dca0 auth_method 0 2024-03-29T11:42:18.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.271+0000 7fe61bfff640 10 client.4535.objecter ms_handle_connect 0x7fe62006dca0 2024-03-29T11:42:18.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.277+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe62c050970 con 0x7fe63413ee90 2024-03-29T11:42:18.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.277+0000 7fe61bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:18.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.277+0000 7fe61bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:18.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.600+0000 7fe639ab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:18.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.600+0000 7fe639ab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:18.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.600+0000 7fe639ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:18.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.600+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6340b2530 con 0x7fe63413ee90 2024-03-29T11:42:18.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.600+0000 7fe61bfff640 1 -- 172.21.15.33:0/272427872 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fe62c0528a0 con 0x7fe63413ee90 2024-03-29T11:42:18.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.600+0000 7fe61bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:18.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.600+0000 7fe61bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:18.605 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:18.605 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:18.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 10 librados: watch_flush enter 2024-03-29T11:42:18.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 10 librados: watch_flush exit 2024-03-29T11:42:18.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:42:18.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:42:18.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 >> v1:172.21.15.153:6816/2737028074 conn(0x7fe62006dca0 legacy=0x7fe620070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 10 monclient: shutdown 2024-03-29T11:42:18.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:18.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 >> v1:172.21.15.33:6790/0 conn(0x7fe63413ee90 legacy=0x7fe6341348a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:18.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 shutdown_connections 2024-03-29T11:42:18.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.604+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 >> 172.21.15.33:0/272427872 conn(0x7fe6340b35e0 msgr2=0x7fe63413b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:18.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.606+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 shutdown_connections 2024-03-29T11:42:18.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.606+0000 7fe639ab3640 1 -- 172.21.15.33:0/272427872 wait complete. 2024-03-29T11:42:18.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:18.606+0000 7fe639ab3640 1 librados: shutdown 2024-03-29T11:42:18.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:25.434 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:25.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.579+0000 7f877dc84640 10 monclient: get_monmap_and_config 2024-03-29T11:42:25.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.579+0000 7f877dc84640 10 monclient: build_initial_monmap 2024-03-29T11:42:25.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.579+0000 7f877dc84640 10 monclient: monmap: 2024-03-29T11:42:25.583 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:25.581544+0000 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:25.581544+0000 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:25.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.581+0000 7f877dc84640 1 Processor -- start 2024-03-29T11:42:25.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.581+0000 7f877dc84640 1 -- start start 2024-03-29T11:42:25.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.581+0000 7f877dc84640 10 monclient: init 2024-03-29T11:42:25.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:25.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:25.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 10 monclient(hunting): picked mon.noname-b con 0x7f87780ae490 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:25.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 10 monclient(hunting): picked mon.noname-a con 0x7f87780aef70 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:25.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 10 monclient(hunting): picked mon.noname-c con 0x7f87780727a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:25.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87780b0020 con 0x7f87780aef70 2024-03-29T11:42:25.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877806d960 con 0x7f87780727a0 2024-03-29T11:42:25.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87780ad310 con 0x7f87780ae490 2024-03-29T11:42:25.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:25.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f877dc84640 10 monclient(hunting): authenticate will time out at 1301.847028s 2024-03-29T11:42:25.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f8777fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f87780aef70 0x7f8778072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:60678/0 (socket says 172.21.15.33:60678) 2024-03-29T11:42:25.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.582+0000 7f8777fff640 1 -- 172.21.15.33:0/3148992230 learned_addr learned my addr 172.21.15.33:0/3148992230 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:25.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631959631 0 0) 0x7f87780b0020 con 0x7f87780aef70 2024-03-29T11:42:25.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8764003520 con 0x7f87780aef70 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461993562 0 0) 0x7f87780ad310 con 0x7f87780ae490 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87780b0020 con 0x7f87780ae490 2024-03-29T11:42:25.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456191589 0 0) 0x7f877806d960 con 0x7f87780727a0 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87780ad310 con 0x7f87780727a0 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3014920536 0 0) 0x7f87780b0020 con 0x7f87780ae490 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 >> v1:172.21.15.153:6789/0 conn(0x7f87780727a0 legacy=0x7f877814b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 >> v1:172.21.15.33:6789/0 conn(0x7f87780aef70 legacy=0x7f8778072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 monclient: found mon.noname-b 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87781432d0 con 0x7f87780ae490 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:25.589 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f877dc84640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f87600015f0 con 0x7f87780ae490 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f877dc84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.583+0000 7f87777fe640 10 monclient: dump: 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:25.590 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:25.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f877dc84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8760001750 con 0x7f87780ae490 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f87777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f87777fe640 1 -- 172.21.15.33:0/3148992230 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8760001b60 con 0x7f87780ae490 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f87777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f87777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f87777fe640 10 monclient: dump: 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:25.594 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f877dc84640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.584+0000 7f877dc84640 4 set_mon_vals no callback set 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.585+0000 7f877dc84640 10 monclient: shutdown 2024-03-29T11:42:25.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.585+0000 7f877dc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.585+0000 7f877dc84640 1 -- 172.21.15.33:0/3148992230 >> v1:172.21.15.33:6790/0 conn(0x7f87780ae490 legacy=0x7f87780713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.585+0000 7f877dc84640 1 -- 172.21.15.33:0/3148992230 shutdown_connections 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.585+0000 7f877dc84640 1 -- 172.21.15.33:0/3148992230 >> 172.21.15.33:0/3148992230 conn(0x7f87780673e0 msgr2=0x7f87780b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.586+0000 7f877dc84640 1 -- 172.21.15.33:0/3148992230 shutdown_connections 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.586+0000 7f877dc84640 1 -- 172.21.15.33:0/3148992230 wait complete. 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 10 monclient: build_initial_monmap 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 10 monclient: monmap: 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:25.596 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 1 librados: starting msgr at 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 1 librados: starting objecter 2024-03-29T11:42:25.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 1 Processor -- start 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 1 -- start start 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 1 librados: setting wanted keys 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 1 librados: calling monclient init 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.587+0000 7f877dc84640 10 monclient: init 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.588+0000 7f877dc84640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.588+0000 7f877dc84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.588+0000 7f877dc84640 10 monclient(hunting): picked mon.noname-c con 0x7f87780727a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.588+0000 7f877dc84640 10 monclient(hunting): picked mon.noname-a con 0x7f87780ae490 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.588+0000 7f877dc84640 10 monclient(hunting): picked mon.noname-b con 0x7f87780aef70 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f8777fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f87780ae490 0x7f8778070a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:33118/0 (socket says 172.21.15.33:33118) 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f877cc82640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f87780727a0 0x7f877806dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:54832/0 (socket says 172.21.15.33:54832) 2024-03-29T11:42:25.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f8777fff640 1 -- 172.21.15.33:0/2683428839 learned_addr learned my addr 172.21.15.33:0/2683428839 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8778070260 con 0x7f87780ae490 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8778070500 con 0x7f87780aef70 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877806ebb0 con 0x7f87780727a0 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f877dc84640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 client.?.objecter ms_handle_connect 0x7f87780727a0 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f877dc84640 10 monclient(hunting): authenticate will time out at 1301.853806s 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 client.?.objecter ms_handle_connect 0x7f87780ae490 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503703431 0 0) 0x7f8778070260 con 0x7f87780ae490 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8758004eb0 con 0x7f87780ae490 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 client.?.objecter ms_handle_connect 0x7f87780aef70 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.589+0000 7f87757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683641486 0 0) 0x7f877806ebb0 con 0x7f87780727a0 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8778070260 con 0x7f87780727a0 2024-03-29T11:42:25.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500877229 0 0) 0x7f8758004eb0 con 0x7f87780ae490 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f877806ebb0 con 0x7f87780ae490 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242203150 0 0) 0x7f8778070260 con 0x7f87780727a0 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8758004eb0 con 0x7f87780727a0 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f87680018d0 con 0x7f87780ae490 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): dump: 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:25.601 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8760001a60 con 0x7f87780727a0 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): dump: 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:25.602 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:25.603 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:25.603 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:25.603 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032968426 0 0) 0x7f8778070500 con 0x7f87780aef70 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8778070260 con 0x7f87780aef70 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.590+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3666579636 0 0) 0x7f8758004eb0 con 0x7f87780727a0 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 >> v1:172.21.15.153:6789/0 conn(0x7f87780aef70 legacy=0x7f87781ee330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 >> v1:172.21.15.33:6789/0 conn(0x7f87780ae490 legacy=0x7f8778070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:25.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 1 monclient: found mon.c 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877806ed40 con 0x7f87780727a0 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 10 monclient: _renew_subs 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8778140740 con 0x7f87780727a0 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 10 monclient: _renew_subs 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87781efa50 con 0x7f87780727a0 2024-03-29T11:42:25.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f877dc84640 1 librados: init done 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8760001bc0 con 0x7f87780727a0 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8760003a40 con 0x7f87780727a0 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87767fc640 4 set_mon_vals no callback set 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.591+0000 7f87757fa640 10 monclient: dump: 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:25.607 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.592+0000 7f877dc84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.592+0000 7f877dc84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.592+0000 7f877dc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.592+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875c005080 con 0x7f87780727a0 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.594+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f8760003ba0 con 0x7f87780727a0 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.594+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f8760084990 con 0x7f87780727a0 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.594+0000 7f87757fa640 10 client.4541.objecter ms_dispatch 0x7f8778000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.594+0000 7f87757fa640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.594+0000 7f87757fa640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.594+0000 7f87757fa640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.595+0000 7f8777fff640 10 monclient: get_auth_request con 0x7f875806df40 auth_method 0 2024-03-29T11:42:25.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.595+0000 7f87757fa640 10 client.4541.objecter ms_handle_connect 0x7f875806df40 2024-03-29T11:42:25.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.602+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8760050270 con 0x7f87780727a0 2024-03-29T11:42:25.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.602+0000 7f87757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:25.609 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.602+0000 7f87757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:26.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.997+0000 7f877dc84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:26.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.997+0000 7f877dc84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:26.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.997+0000 7f877dc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:26.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.997+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f875c005330 con 0x7f87780727a0 2024-03-29T11:42:26.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.998+0000 7f87757fa640 1 -- 172.21.15.33:0/2683428839 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f87600521a0 con 0x7f87780727a0 2024-03-29T11:42:26.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.998+0000 7f87757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:26.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:25.998+0000 7f87757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:26.002 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:26.002 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:26.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 10 librados: watch_flush enter 2024-03-29T11:42:26.005 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 10 librados: watch_flush exit 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 >> v1:172.21.15.153:6816/2737028074 conn(0x7f875806df40 legacy=0x7f8758070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 10 monclient: shutdown 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.001+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 >> v1:172.21.15.33:6790/0 conn(0x7f87780727a0 legacy=0x7f877806dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.002+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 shutdown_connections 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.002+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 >> 172.21.15.33:0/2683428839 conn(0x7f87780673e0 msgr2=0x7f8778072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.002+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 shutdown_connections 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.002+0000 7f877dc84640 1 -- 172.21.15.33:0/2683428839 wait complete. 2024-03-29T11:42:26.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:26.002+0000 7f877dc84640 1 librados: shutdown 2024-03-29T11:42:26.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:32.829 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.966+0000 7f2bcc44e640 10 monclient: get_monmap_and_config 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.966+0000 7f2bcc44e640 10 monclient: build_initial_monmap 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.966+0000 7f2bcc44e640 10 monclient: monmap: 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:32.966911+0000 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:32.966911+0000 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:32.969 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:32.970 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:32.970 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:32.970 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:32.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.967+0000 7f2bcc44e640 1 Processor -- start 2024-03-29T11:42:32.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.967+0000 7f2bcc44e640 1 -- start start 2024-03-29T11:42:32.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.967+0000 7f2bcc44e640 10 monclient: init 2024-03-29T11:42:32.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:32.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:32.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc41457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:32.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc413f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:32.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc40a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc40aaef0 con 0x7f2bc41457f0 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4145bd0 con 0x7f2bc413f650 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4145eb0 con 0x7f2bc40a9d70 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.968+0000 7f2bcc44e640 10 monclient(hunting): authenticate will time out at 1309.232541s 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.969+0000 7f2bc99e9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2bc413f650 0x7f2bc40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:59334/0 (socket says 172.21.15.33:59334) 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.969+0000 7f2bc99e9640 1 -- 172.21.15.33:0/3376640938 learned_addr learned my addr 172.21.15.33:0/3376640938 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.969+0000 7f2bc99e9640 1 -- 172.21.15.33:0/3376640938 >> v1:172.21.15.153:6789/0 conn(0x7f2bc413f650 legacy=0x7f2bc40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.969+0000 7f2bc99e9640 1 -- 172.21.15.33:0/3376640938 >> v1:172.21.15.153:6789/0 conn(0x7f2bc413f650 legacy=0x7f2bc40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:42:32.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.969+0000 7f2bc99e9640 1 --1- 172.21.15.33:0/3376640938 >> v1:172.21.15.153:6789/0 conn(0x7f2bc413f650 0x7f2bc40b2620 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744646955 0 0) 0x7f2bc4145eb0 con 0x7f2bc40a9d70 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 10 monclient(hunting): _negotiate 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb0003520 con 0x7f2bc40a9d70 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1008561116 0 0) 0x7f2bb0003520 con 0x7f2bc40a9d70 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 >> v1:172.21.15.153:6789/0 conn(0x7f2bc413f650 legacy=0x7f2bc40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:32.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.970+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 >> v1:172.21.15.33:6789/0 conn(0x7f2bc41457f0 legacy=0x7f2bc413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:32.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 10 monclient: _finish_hunting 0 2024-03-29T11:42:32.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 1 monclient: found mon.noname-b 2024-03-29T11:42:32.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:32.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:42:32.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc4146340 con 0x7f2bc40a9d70 2024-03-29T11:42:32.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 10 monclient: _finish_auth 0 2024-03-29T11:42:32.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 10 monclient: _check_auth_tickets 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2bbc001a10 con 0x7f2bc40a9d70 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.971+0000 7f2bc91e8640 10 monclient: dump: 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:32.976 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bcc44e640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bcc44e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bbc001c70 con 0x7f2bc40a9d70 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bc91e8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bc91e8640 1 -- 172.21.15.33:0/3376640938 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2bbc0020c0 con 0x7f2bc40a9d70 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bc91e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bc91e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bc91e8640 10 monclient: dump: 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:32.977 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:32.978 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:32.978 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:32.978 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bcc44e640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:32.978 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.972+0000 7f2bcc44e640 4 set_mon_vals no callback set 2024-03-29T11:42:32.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.973+0000 7f2bcc44e640 10 monclient: shutdown 2024-03-29T11:42:32.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.973+0000 7f2bcc44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:32.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.973+0000 7f2bcc44e640 1 -- 172.21.15.33:0/3376640938 >> v1:172.21.15.33:6790/0 conn(0x7f2bc40a9d70 legacy=0x7f2bc414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:32.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.973+0000 7f2bcc44e640 1 -- 172.21.15.33:0/3376640938 shutdown_connections 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.973+0000 7f2bcc44e640 1 -- 172.21.15.33:0/3376640938 >> 172.21.15.33:0/3376640938 conn(0x7f2bc4066a00 msgr2=0x7f2bc4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.974+0000 7f2bcc44e640 1 -- 172.21.15.33:0/3376640938 shutdown_connections 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.974+0000 7f2bcc44e640 1 -- 172.21.15.33:0/3376640938 wait complete. 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.974+0000 7f2bcc44e640 10 monclient: build_initial_monmap 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.974+0000 7f2bcc44e640 10 monclient: monmap: 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:32.987 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.975+0000 7f2bcc44e640 1 librados: starting msgr at 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.975+0000 7f2bcc44e640 1 librados: starting objecter 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.975+0000 7f2bcc44e640 1 Processor -- start 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.975+0000 7f2bcc44e640 1 -- start start 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.975+0000 7f2bcc44e640 1 librados: setting wanted keys 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.975+0000 7f2bcc44e640 1 librados: calling monclient init 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.975+0000 7f2bcc44e640 10 monclient: init 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc40a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc413f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc41457f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc413dfe0 con 0x7f2bc40a9d70 2024-03-29T11:42:32.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4130aa0 con 0x7f2bc413f650 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc413e7c0 con 0x7f2bc41457f0 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bcc44e640 10 monclient(hunting): authenticate will time out at 1309.240432s 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bc99e9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2bc413f650 0x7f2bc413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:59414/0 (socket says 172.21.15.33:59414) 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.976+0000 7f2bc99e9640 1 -- 172.21.15.33:0/4020106421 learned_addr learned my addr 172.21.15.33:0/4020106421 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.977+0000 7f2bbaffd640 10 client.?.objecter ms_handle_connect 0x7f2bc413f650 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.977+0000 7f2bbaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.977+0000 7f2bbaffd640 10 client.?.objecter ms_handle_connect 0x7f2bc41457f0 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.977+0000 7f2bbaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88845837 0 0) 0x7f2bc413e7c0 con 0x7f2bc41457f0 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:42:32.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba8004eb0 con 0x7f2bc41457f0 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3081847768 0 0) 0x7f2ba8004eb0 con 0x7f2bc41457f0 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bc413e7c0 con 0x7f2bc41457f0 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2bbc001830 con 0x7f2bc41457f0 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): dump: 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:32.990 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2278872335 0 0) 0x7f2bc413e7c0 con 0x7f2bc41457f0 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 >> v1:172.21.15.153:6789/0 conn(0x7f2bc413f650 legacy=0x7f2bc413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.978+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 >> v1:172.21.15.33:6789/0 conn(0x7f2bc40a9d70 legacy=0x7f2bc4132150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 1 monclient: found mon.c 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc413e950 con 0x7f2bc41457f0 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 10 monclient: _renew_subs 2024-03-29T11:42:32.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc4142b80 con 0x7f2bc41457f0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 10 monclient: _renew_subs 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc4142e80 con 0x7f2bc41457f0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.979+0000 7f2bcc44e640 1 librados: init done 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.981+0000 7f2bcc44e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.981+0000 7f2bcc44e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.981+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bbc003d40 con 0x7f2bc41457f0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bcc44e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4075f10 con 0x7f2bc41457f0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2bbc004050 con 0x7f2bc41457f0 2024-03-29T11:42:32.992 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bbaffd640 10 monclient: dump: 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bc89e7640 4 set_mon_vals no callback set 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.982+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f2bbc01a020 con 0x7f2bc41457f0 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.983+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f2bbc083e00 con 0x7f2bc41457f0 2024-03-29T11:42:32.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.983+0000 7f2bbaffd640 10 client.4547.objecter ms_dispatch 0x7f2bc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.983+0000 7f2bbaffd640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.983+0000 7f2bbaffd640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.983+0000 7f2bbaffd640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.983+0000 7f2bca1ea640 10 monclient: get_auth_request con 0x7f2ba806dea0 auth_method 0 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.984+0000 7f2bbaffd640 10 client.4547.objecter ms_handle_connect 0x7f2ba806dea0 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.990+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bbc04f790 con 0x7f2bc41457f0 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.990+0000 7f2bbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:32.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:32.990+0000 7f2bbaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:33.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.297+0000 7f2bcc44e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:33.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.297+0000 7f2bcc44e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:33.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.297+0000 7f2bcc44e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:33.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.297+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bc404ef70 con 0x7f2bc41457f0 2024-03-29T11:42:33.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.298+0000 7f2bbaffd640 1 -- 172.21.15.33:0/4020106421 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f2bbc0516c0 con 0x7f2bc41457f0 2024-03-29T11:42:33.301 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.298+0000 7f2bbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:33.301 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.298+0000 7f2bbaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:33.301 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:33.301 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:33.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 10 librados: watch_flush enter 2024-03-29T11:42:33.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 10 librados: watch_flush exit 2024-03-29T11:42:33.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:42:33.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 >> v1:172.21.15.153:6816/2737028074 conn(0x7f2ba806dea0 legacy=0x7f2ba8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 10 monclient: shutdown 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 >> v1:172.21.15.33:6790/0 conn(0x7f2bc41457f0 legacy=0x7f2bc41e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.301+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 shutdown_connections 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.302+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 >> 172.21.15.33:0/4020106421 conn(0x7f2bc4066a00 msgr2=0x7f2bc40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.302+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 shutdown_connections 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.302+0000 7f2bcc44e640 1 -- 172.21.15.33:0/4020106421 wait complete. 2024-03-29T11:42:33.305 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:33.302+0000 7f2bcc44e640 1 librados: shutdown 2024-03-29T11:42:33.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:40.125 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.266+0000 7fbab70e6640 10 monclient: get_monmap_and_config 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.266+0000 7fbab70e6640 10 monclient: build_initial_monmap 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.266+0000 7fbab70e6640 10 monclient: monmap: 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:40.267528+0000 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:40.267528+0000 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:40.270 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:40.271 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:40.271 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:40.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.268+0000 7fbab70e6640 1 Processor -- start 2024-03-29T11:42:40.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.268+0000 7fbab70e6640 1 -- start start 2024-03-29T11:42:40.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.268+0000 7fbab70e6640 10 monclient: init 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 10 monclient(hunting): picked mon.noname-b con 0x7fbab00b4100 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbab013ee90 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 10 monclient(hunting): picked mon.noname-c con 0x7fbab00af900 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab00afd10 con 0x7fbab013ee90 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab00af6f0 con 0x7fbab00af900 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab00aac80 con 0x7fbab00b4100 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.269+0000 7fbab70e6640 10 monclient(hunting): authenticate will time out at 1316.533330s 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.270+0000 7fbab5683640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbab00af900 0x7fbab014be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:44614/0 (socket says 172.21.15.33:44614) 2024-03-29T11:42:40.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.270+0000 7fbab5683640 1 -- 172.21.15.33:0/373193197 learned_addr learned my addr 172.21.15.33:0/373193197 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:40.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.272+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944048590 0 0) 0x7fbab00af6f0 con 0x7fbab00af900 2024-03-29T11:42:40.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.272+0000 7fbaaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.272+0000 7fbaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.272+0000 7fbaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.272+0000 7fbaaf7fe640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:42:40.275 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.272+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa0003520 con 0x7fbab00af900 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803705873 0 0) 0x7fbab00aac80 con 0x7fbab00b4100 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab00af6f0 con 0x7fbab00b4100 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778356668 0 0) 0x7fbab00afd10 con 0x7fbab013ee90 2024-03-29T11:42:40.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab00aac80 con 0x7fbab013ee90 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3424009346 0 0) 0x7fbab00aac80 con 0x7fbab013ee90 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 >> v1:172.21.15.33:6790/0 conn(0x7fbab00b4100 legacy=0x7fbab00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 >> v1:172.21.15.153:6789/0 conn(0x7fbab00af900 legacy=0x7fbab014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 1 monclient: found mon.noname-a 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.273+0000 7fbaaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab00aaf60 con 0x7fbab013ee90 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:40.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fba9c0015b0 con 0x7fbab013ee90 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 10 monclient: dump: 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:40.279 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:40.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbab70e6640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:42:40.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbab70e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 1 -- 172.21.15.33:0/373193197 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba9c001710 con 0x7fbab013ee90 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbaaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbab70e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.274+0000 7fbab70e6640 4 set_mon_vals no callback set 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.275+0000 7fbab70e6640 10 monclient: shutdown 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.275+0000 7fbab70e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.275+0000 7fbab70e6640 1 -- 172.21.15.33:0/373193197 >> v1:172.21.15.33:6789/0 conn(0x7fbab013ee90 legacy=0x7fbab013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.275+0000 7fbab70e6640 1 -- 172.21.15.33:0/373193197 shutdown_connections 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.275+0000 7fbab70e6640 1 -- 172.21.15.33:0/373193197 >> 172.21.15.33:0/373193197 conn(0x7fbab00b35e0 msgr2=0x7fbab00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.275+0000 7fbab70e6640 1 -- 172.21.15.33:0/373193197 shutdown_connections 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.275+0000 7fbab70e6640 1 -- 172.21.15.33:0/373193197 wait complete. 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.276+0000 7fbab70e6640 10 monclient: build_initial_monmap 2024-03-29T11:42:40.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.276+0000 7fbab70e6640 10 monclient: monmap: 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.277+0000 7fbab70e6640 1 librados: starting msgr at 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.277+0000 7fbab70e6640 1 librados: starting objecter 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.277+0000 7fbab70e6640 1 Processor -- start 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.277+0000 7fbab70e6640 1 -- start start 2024-03-29T11:42:40.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.277+0000 7fbab70e6640 1 librados: setting wanted keys 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.277+0000 7fbab70e6640 1 librados: calling monclient init 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.277+0000 7fbab70e6640 10 monclient: init 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 10 monclient(hunting): picked mon.noname-c con 0x7fbab00af900 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbab00b4100 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 10 monclient(hunting): picked mon.noname-b con 0x7fbab013ee90 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab012b120 con 0x7fbab00b4100 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab013d230 con 0x7fbab013ee90 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab013da90 con 0x7fbab00af900 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:40.285 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.278+0000 7fbab70e6640 10 monclient(hunting): authenticate will time out at 1316.542493s 2024-03-29T11:42:40.286 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.281+0000 7fbab5683640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbab013ee90 0x7fbab01e26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:44738/0 (socket says 172.21.15.33:44738) 2024-03-29T11:42:40.286 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.281+0000 7fbab5683640 1 -- 172.21.15.33:0/2323905656 learned_addr learned my addr 172.21.15.33:0/2323905656 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:40.286 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.283+0000 7fbaad7fa640 10 client.?.objecter ms_handle_connect 0x7fbab013ee90 2024-03-29T11:42:40.286 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.283+0000 7fbaad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.283+0000 7fbaad7fa640 10 client.?.objecter ms_handle_connect 0x7fbab00b4100 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.283+0000 7fbaad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.283+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859679313 0 0) 0x7fbab012b120 con 0x7fbab00b4100 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.283+0000 7fbaad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.283+0000 7fbaad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba94004eb0 con 0x7fbab00b4100 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbab4e82640 1 -- 172.21.15.33:0/2323905656 >> v1:172.21.15.33:6790/0 conn(0x7fbab00af900 legacy=0x7fbab013c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbab4e82640 1 --1- 172.21.15.33:0/2323905656 >> v1:172.21.15.33:6790/0 conn(0x7fbab00af900 0x7fbab013c0c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T11:42:40.287 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.33:6790/0 2024-03-29T11:42:40.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 405948196 0 0) 0x7fba94004eb0 con 0x7fbab00b4100 2024-03-29T11:42:40.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbab012b120 con 0x7fbab00b4100 2024-03-29T11:42:40.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fba9c003790 con 0x7fbab00b4100 2024-03-29T11:42:40.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:40.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:40.289 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893262835 0 0) 0x7fbab013d230 con 0x7fbab013ee90 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:40.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.284+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba94004eb0 con 0x7fbab013ee90 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3868278843 0 0) 0x7fbab012b120 con 0x7fbab00b4100 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 >> v1:172.21.15.33:6790/0 conn(0x7fbab00af900 legacy=0x7fbab013c0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 >> v1:172.21.15.153:6789/0 conn(0x7fbab013ee90 legacy=0x7fbab01e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 1 monclient: found mon.a 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab013dc20 con 0x7fbab00b4100 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbaad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:42:40.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 10 monclient: _renew_subs 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab013f5c0 con 0x7fbab00b4100 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 10 monclient: _renew_subs 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab013faf0 con 0x7fbab00b4100 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.285+0000 7fbab70e6640 1 librados: init done 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.286+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba9c0019c0 con 0x7fbab00b4100 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.286+0000 7fbaad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.286+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fba9c003d40 con 0x7fbab00b4100 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.286+0000 7fbaad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.286+0000 7fbaad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.286+0000 7fbaad7fa640 10 monclient: dump: 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:40.292 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:40.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.286+0000 7fbaaeffd640 4 set_mon_vals no callback set 2024-03-29T11:42:40.294 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.288+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fba9c003ea0 con 0x7fbab00b4100 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.289+0000 7fbab70e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.289+0000 7fbab70e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.289+0000 7fbab70e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.289+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba90005080 con 0x7fbab00b4100 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.290+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fba9c083900 con 0x7fbab00b4100 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.290+0000 7fbaad7fa640 10 client.4500.objecter ms_dispatch 0x7fbab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.290+0000 7fbaad7fa640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.290+0000 7fbaad7fa640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:40.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.290+0000 7fbaad7fa640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:42:40.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.293+0000 7fbab4e82640 10 monclient: get_auth_request con 0x7fba9406db20 auth_method 0 2024-03-29T11:42:40.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.293+0000 7fbaad7fa640 10 client.4500.objecter ms_handle_connect 0x7fba9406db20 2024-03-29T11:42:40.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.298+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba9c04f320 con 0x7fbab00b4100 2024-03-29T11:42:40.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.298+0000 7fbaad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:40.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.298+0000 7fbaad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:40.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.622+0000 7fbab70e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:40.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.622+0000 7fbab70e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:40.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.622+0000 7fbab70e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:40.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.622+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba90005330 con 0x7fbab00b4100 2024-03-29T11:42:40.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.624+0000 7fbaad7fa640 1 -- 172.21.15.33:0/2323905656 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fba9c051250 con 0x7fbab00b4100 2024-03-29T11:42:40.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.624+0000 7fbaad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:40.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.624+0000 7fbaad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:40.895 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:40.895 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 10 librados: watch_flush enter 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 10 librados: watch_flush exit 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 >> v1:172.21.15.153:6816/2737028074 conn(0x7fba9406db20 legacy=0x7fba9406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 10 monclient: shutdown 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 >> v1:172.21.15.33:6789/0 conn(0x7fbab00b4100 legacy=0x7fbab012c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 shutdown_connections 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 >> 172.21.15.33:0/2323905656 conn(0x7fbab00b35e0 msgr2=0x7fbab0143140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:40.897 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 shutdown_connections 2024-03-29T11:42:40.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.632+0000 7fbab70e6640 1 -- 172.21.15.33:0/2323905656 wait complete. 2024-03-29T11:42:40.898 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:40.633+0000 7fbab70e6640 1 librados: shutdown 2024-03-29T11:42:40.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:47.704 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:47.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.855+0000 7f3e77fff640 10 monclient: get_monmap_and_config 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.855+0000 7f3e77fff640 10 monclient: build_initial_monmap 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.855+0000 7f3e77fff640 10 monclient: monmap: 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:47.856458+0000 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:47.856458+0000 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:47.859 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:47.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.856+0000 7f3e77fff640 1 Processor -- start 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.856+0000 7f3e77fff640 1 -- start start 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.856+0000 7f3e77fff640 10 monclient: init 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 10 monclient(hunting): picked mon.noname-a con 0x7f3e70006a70 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7002b100 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 10 monclient(hunting): picked mon.noname-c con 0x7f3e70029c30 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7002a270 con 0x7f3e70006a70 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7002c440 con 0x7f3e70029c30 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e700c9440 con 0x7f3e7002b100 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:47.861 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.857+0000 7f3e77fff640 10 monclient(hunting): authenticate will time out at 1324.121469s 2024-03-29T11:42:47.866 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e76ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f3e70006a70 0x7f3e700c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:57361/0 (socket says 172.21.15.33:57361) 2024-03-29T11:42:47.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e76ffd640 1 -- 172.21.15.33:0/3229676021 learned_addr learned my addr 172.21.15.33:0/3229676021 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:47.867 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e767fc640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f3e7002b100 0x7f3e700cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:55465/0 (socket says 172.21.15.33:55465) 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992892071 0 0) 0x7f3e7002a270 con 0x7f3e70006a70 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e60003520 con 0x7f3e70006a70 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169048400 0 0) 0x7f3e700c9440 con 0x7f3e7002b100 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e7002a270 con 0x7f3e7002b100 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.864+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489007039 0 0) 0x7f3e7002c440 con 0x7f3e70029c30 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e700c9440 con 0x7f3e70029c30 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627528338 0 0) 0x7f3e60003520 con 0x7f3e70006a70 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 >> v1:172.21.15.33:6790/0 conn(0x7f3e7002b100 legacy=0x7f3e700cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 >> v1:172.21.15.153:6789/0 conn(0x7f3e70029c30 legacy=0x7f3e700d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 1 monclient: found mon.noname-a 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:42:47.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e700c95d0 con 0x7f3e70006a70 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3e7804f730 con 0x7f3e70006a70 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.865+0000 7f3e75ffb640 10 monclient: dump: 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:47.872 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.866+0000 7f3e77fff640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.866+0000 7f3e77fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.866+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e78075f10 con 0x7f3e70006a70 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.866+0000 7f3e75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.866+0000 7f3e77fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.866+0000 7f3e77fff640 4 set_mon_vals no callback set 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.867+0000 7f3e75ffb640 1 -- 172.21.15.33:0/3229676021 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3e780758b0 con 0x7f3e70006a70 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.867+0000 7f3e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.867+0000 7f3e75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.867+0000 7f3e75ffb640 10 monclient: dump: 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:47.873 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.867+0000 7f3e77fff640 10 monclient: shutdown 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.867+0000 7f3e77fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:47.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.867+0000 7f3e77fff640 1 -- 172.21.15.33:0/3229676021 >> v1:172.21.15.33:6789/0 conn(0x7f3e70006a70 legacy=0x7f3e700c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.868+0000 7f3e77fff640 1 -- 172.21.15.33:0/3229676021 shutdown_connections 2024-03-29T11:42:47.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.868+0000 7f3e77fff640 1 -- 172.21.15.33:0/3229676021 >> 172.21.15.33:0/3229676021 conn(0x7f3e700058f0 msgr2=0x7f3e70005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:47.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.871+0000 7f3e77fff640 1 -- 172.21.15.33:0/3229676021 shutdown_connections 2024-03-29T11:42:47.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.871+0000 7f3e77fff640 1 -- 172.21.15.33:0/3229676021 wait complete. 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.872+0000 7f3e77fff640 10 monclient: build_initial_monmap 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.872+0000 7f3e77fff640 10 monclient: monmap: 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.872+0000 7f3e77fff640 1 librados: starting msgr at 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.872+0000 7f3e77fff640 1 librados: starting objecter 2024-03-29T11:42:47.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.872+0000 7f3e77fff640 1 Processor -- start 2024-03-29T11:42:47.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.873+0000 7f3e77fff640 1 -- start start 2024-03-29T11:42:47.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.873+0000 7f3e77fff640 1 librados: setting wanted keys 2024-03-29T11:42:47.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.873+0000 7f3e77fff640 1 librados: calling monclient init 2024-03-29T11:42:47.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.873+0000 7f3e77fff640 10 monclient: init 2024-03-29T11:42:47.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:47.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:47.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 10 monclient(hunting): picked mon.noname-c con 0x7f3e70006a70 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:47.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 10 monclient(hunting): picked mon.noname-a con 0x7f3e70029c30 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:47.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7002c9a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:47.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7016c900 con 0x7f3e70029c30 2024-03-29T11:42:47.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e700caad0 con 0x7f3e7002c9a0 2024-03-29T11:42:47.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7002f070 con 0x7f3e70006a70 2024-03-29T11:42:47.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:47.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.874+0000 7f3e77fff640 10 monclient(hunting): authenticate will time out at 1324.138258s 2024-03-29T11:42:47.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.880+0000 7f3e777fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3e7002c9a0 0x7f3e701664d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:52823/0 (socket says 172.21.15.33:52823) 2024-03-29T11:42:47.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.880+0000 7f3e777fe640 1 -- 172.21.15.33:0/2408659230 learned_addr learned my addr 172.21.15.33:0/2408659230 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:47.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.880+0000 7f3e5ffff640 10 client.?.objecter ms_handle_connect 0x7f3e7002c9a0 2024-03-29T11:42:47.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.880+0000 7f3e5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 client.?.objecter ms_handle_connect 0x7f3e70029c30 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 client.?.objecter ms_handle_connect 0x7f3e70006a70 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009373814 0 0) 0x7f3e700caad0 con 0x7f3e7002c9a0 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e64004eb0 con 0x7f3e7002c9a0 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558324033 0 0) 0x7f3e7016c900 con 0x7f3e70029c30 2024-03-29T11:42:47.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.881+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e700caad0 con 0x7f3e70029c30 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130146834 0 0) 0x7f3e7002f070 con 0x7f3e70006a70 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e7016c900 con 0x7f3e70006a70 2024-03-29T11:42:47.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1252554068 0 0) 0x7f3e700caad0 con 0x7f3e70029c30 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e7002f070 con 0x7f3e70029c30 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3e6c001890 con 0x7f3e70029c30 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:47.887 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.882+0000 7f3e5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.883+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1447846453 0 0) 0x7f3e7016c900 con 0x7f3e70006a70 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.883+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e700caad0 con 0x7f3e70006a70 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.883+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3e7804f400 con 0x7f3e70006a70 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.883+0000 7f3e5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.883+0000 7f3e5ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.883+0000 7f3e5ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:47.888 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2881609573 0 0) 0x7f3e700caad0 con 0x7f3e70006a70 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 >> v1:172.21.15.153:6789/0 conn(0x7f3e7002c9a0 legacy=0x7f3e701664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 >> v1:172.21.15.33:6789/0 conn(0x7f3e70029c30 legacy=0x7f3e7016cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 1 monclient: found mon.c 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:47.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7002dca0 con 0x7f3e70006a70 2024-03-29T11:42:47.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:47.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:47.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e77fff640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e77fff640 10 monclient: _renew_subs 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.885+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7016d4a0 con 0x7f3e70006a70 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e77fff640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e77fff640 10 monclient: _renew_subs 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e70166be0 con 0x7f3e70006a70 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e77fff640 1 librados: init done 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e7804fdd0 con 0x7f3e70006a70 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:47.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.886+0000 7f3e74ff9640 4 set_mon_vals no callback set 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.888+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f3e78067bf0 con 0x7f3e70006a70 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.888+0000 7f3e5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.888+0000 7f3e5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.888+0000 7f3e5ffff640 10 monclient: dump: 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:47.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.888+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f3e78067e10 con 0x7f3e70006a70 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.889+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f3e780ff360 con 0x7f3e70006a70 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.889+0000 7f3e5ffff640 10 client.4556.objecter ms_dispatch 0x7f3e78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.889+0000 7f3e5ffff640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.889+0000 7f3e5ffff640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.889+0000 7f3e5ffff640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.892+0000 7f3e77fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.892+0000 7f3e77fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:47.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.892+0000 7f3e77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:47.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.892+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e58005080 con 0x7f3e70006a70 2024-03-29T11:42:47.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.892+0000 7f3e767fc640 10 monclient: get_auth_request con 0x7f3e6406de10 auth_method 0 2024-03-29T11:42:47.896 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.893+0000 7f3e5ffff640 10 client.4556.objecter ms_handle_connect 0x7f3e6406de10 2024-03-29T11:42:47.902 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.900+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e780cad20 con 0x7f3e70006a70 2024-03-29T11:42:47.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.900+0000 7f3e5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:47.903 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:47.900+0000 7f3e5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:48.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.224+0000 7f3e77fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:48.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.224+0000 7f3e77fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:48.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.224+0000 7f3e77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:42:48.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.224+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e58005330 con 0x7f3e70006a70 2024-03-29T11:42:48.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.227+0000 7f3e5ffff640 1 -- 172.21.15.33:0/2408659230 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f3e780ccc50 con 0x7f3e70006a70 2024-03-29T11:42:48.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.227+0000 7f3e5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:48.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.227+0000 7f3e5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:48.230 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:48.230 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:48.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 10 librados: watch_flush enter 2024-03-29T11:42:48.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 10 librados: watch_flush exit 2024-03-29T11:42:48.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:42:48.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:42:48.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 >> v1:172.21.15.153:6816/2737028074 conn(0x7f3e6406de10 legacy=0x7f3e640702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:48.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 10 monclient: shutdown 2024-03-29T11:42:48.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:48.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.230+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 >> v1:172.21.15.33:6790/0 conn(0x7f3e70006a70 legacy=0x7f3e7002d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:48.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.232+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 shutdown_connections 2024-03-29T11:42:48.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.232+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 >> 172.21.15.33:0/2408659230 conn(0x7f3e700058f0 msgr2=0x7f3e70034d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:48.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.234+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 shutdown_connections 2024-03-29T11:42:48.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.234+0000 7f3e77fff640 1 -- 172.21.15.33:0/2408659230 wait complete. 2024-03-29T11:42:48.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:48.234+0000 7f3e77fff640 1 librados: shutdown 2024-03-29T11:42:48.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:55.060 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.209+0000 7f2d9d104640 10 monclient: get_monmap_and_config 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.209+0000 7f2d9d104640 10 monclient: build_initial_monmap 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.209+0000 7f2d9d104640 10 monclient: monmap: 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:42:55.209698+0000 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:42:55.209698+0000 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:55.213 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:42:55.214 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:42:55.214 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:55.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.210+0000 7f2d9d104640 1 Processor -- start 2024-03-29T11:42:55.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.210+0000 7f2d9d104640 1 -- start start 2024-03-29T11:42:55.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.210+0000 7f2d9d104640 10 monclient: init 2024-03-29T11:42:55.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:55.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:55.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 10 monclient(hunting): picked mon.noname-a con 0x7f2d980a9ac0 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:55.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 10 monclient(hunting): picked mon.noname-c con 0x7f2d9813ee90 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:55.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 10 monclient(hunting): picked mon.noname-b con 0x7f2d980af900 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:55.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980afd10 con 0x7f2d980a9ac0 2024-03-29T11:42:55.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980af6f0 con 0x7f2d9813ee90 2024-03-29T11:42:55.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980aac80 con 0x7f2d980af900 2024-03-29T11:42:55.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:55.216 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.211+0000 7f2d9d104640 10 monclient(hunting): authenticate will time out at 1331.475151s 2024-03-29T11:42:55.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.215+0000 7f2d9759e640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f2d980af900 0x7f2d98149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:36043/0 (socket says 172.21.15.33:36043) 2024-03-29T11:42:55.218 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.216+0000 7f2d9759e640 1 -- 172.21.15.33:0/380491932 learned_addr learned my addr 172.21.15.33:0/380491932 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:55.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.219+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197268143 0 0) 0x7f2d980afd10 con 0x7f2d980a9ac0 2024-03-29T11:42:55.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.219+0000 7f2d95d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:42:55.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.219+0000 7f2d95d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:55.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.219+0000 7f2d95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:55.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.219+0000 7f2d95d9b640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:42:55.222 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.219+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d80003520 con 0x7f2d980a9ac0 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3461212515 0 0) 0x7f2d80003520 con 0x7f2d980a9ac0 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 >> v1:172.21.15.33:6790/0 conn(0x7f2d980af900 legacy=0x7f2d98149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 >> v1:172.21.15.153:6789/0 conn(0x7f2d9813ee90 legacy=0x7f2d9813b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 1 monclient: found mon.noname-a 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980aaf60 con 0x7f2d980a9ac0 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: _finish_auth 0 2024-03-29T11:42:55.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d9d104640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d9d104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2d88001690 con 0x7f2d980a9ac0 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: dump: 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:55.227 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d9d104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d88001860 con 0x7f2d980a9ac0 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d95d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d9d104640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.220+0000 7f2d9d104640 4 set_mon_vals no callback set 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.221+0000 7f2d95d9b640 1 -- 172.21.15.33:0/380491932 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2d88001cb0 con 0x7f2d980a9ac0 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.221+0000 7f2d95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.221+0000 7f2d95d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.221+0000 7f2d95d9b640 10 monclient: dump: 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.228 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.222+0000 7f2d9d104640 10 monclient: shutdown 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.222+0000 7f2d9d104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.222+0000 7f2d9d104640 1 -- 172.21.15.33:0/380491932 >> v1:172.21.15.33:6789/0 conn(0x7f2d980a9ac0 legacy=0x7f2d980adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.222+0000 7f2d9d104640 1 -- 172.21.15.33:0/380491932 shutdown_connections 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.222+0000 7f2d9d104640 1 -- 172.21.15.33:0/380491932 >> 172.21.15.33:0/380491932 conn(0x7f2d980b35e0 msgr2=0x7f2d980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.223+0000 7f2d9d104640 1 -- 172.21.15.33:0/380491932 shutdown_connections 2024-03-29T11:42:55.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.223+0000 7f2d9d104640 1 -- 172.21.15.33:0/380491932 wait complete. 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 10 monclient: build_initial_monmap 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 10 monclient: monmap: 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:55.231 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 1 librados: starting msgr at 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 1 librados: starting objecter 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 1 Processor -- start 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 1 -- start start 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 1 librados: setting wanted keys 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 1 librados: calling monclient init 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.224+0000 7f2d9d104640 10 monclient: init 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 10 monclient(hunting): picked mon.noname-c con 0x7f2d980af900 addr v1:172.21.15.33:6790/0 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 10 monclient(hunting): picked mon.noname-a con 0x7f2d9813ee90 addr v1:172.21.15.33:6789/0 2024-03-29T11:42:55.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 10 monclient(hunting): picked mon.noname-b con 0x7f2d981e4b80 addr v1:172.21.15.153:6789/0 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d98141510 con 0x7f2d9813ee90 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d981e43b0 con 0x7f2d981e4b80 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d981e4660 con 0x7f2d980af900 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.225+0000 7f2d9d104640 10 monclient(hunting): authenticate will time out at 1331.489290s 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.228+0000 7f2d9659c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f2d9813ee90 0x7f2d98071cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:36815/0 (socket says 172.21.15.33:36815) 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.228+0000 7f2d9659c640 1 -- 172.21.15.33:0/773652532 learned_addr learned my addr 172.21.15.33:0/773652532 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.228+0000 7f2d7f7fe640 10 client.?.objecter ms_handle_connect 0x7f2d9813ee90 2024-03-29T11:42:55.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.228+0000 7f2d7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:55.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297882851 0 0) 0x7f2d98141510 con 0x7f2d9813ee90 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d84004eb0 con 0x7f2d9813ee90 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1851514974 0 0) 0x7f2d84004eb0 con 0x7f2d9813ee90 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.229+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d84005300 con 0x7f2d9813ee90 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.230+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2d90001bb0 con 0x7f2d9813ee90 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.230+0000 7f2d7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.230+0000 7f2d7f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.230+0000 7f2d7f7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:55.235 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.230+0000 7f2d7f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:55.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.234+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1761381033 0 0) 0x7f2d84005300 con 0x7f2d9813ee90 2024-03-29T11:42:55.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.234+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 >> v1:172.21.15.33:6790/0 conn(0x7f2d980af900 legacy=0x7f2d98141ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:55.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 >> v1:172.21.15.153:6789/0 conn(0x7f2d981e4b80 legacy=0x7f2d9813f5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 1 monclient: found mon.a 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98142600 con 0x7f2d9813ee90 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.236+0000 7f2d7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:55.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 10 monclient: _renew_subs 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d980723d0 con 0x7f2d9813ee90 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 10 monclient: _renew_subs 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9813feb0 con 0x7f2d9813ee90 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.237+0000 7f2d9d104640 1 librados: init done 2024-03-29T11:42:55.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.239+0000 7f2d9d104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.239+0000 7f2d9d104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.239+0000 7f2d9d104640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.239+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d981e48a0 con 0x7f2d9813ee90 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.240+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d90002180 con 0x7f2d9813ee90 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.240+0000 7f2d7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.240+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2d90001bb0 con 0x7f2d9813ee90 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.240+0000 7f2d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.240+0000 7f2d7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.240+0000 7f2d7f7fe640 10 monclient: dump: 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:55.245 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:42:55.246 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:42:55.246 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:42:55.246 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:42:55.246 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:42:55.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.240+0000 7f2d9559a640 4 set_mon_vals no callback set 2024-03-29T11:42:55.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.241+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f2d9001abf0 con 0x7f2d9813ee90 2024-03-29T11:42:55.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.242+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f2d90084930 con 0x7f2d9813ee90 2024-03-29T11:42:55.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.242+0000 7f2d7f7fe640 10 client.4512.objecter ms_dispatch 0x7f2d98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:55.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.242+0000 7f2d7f7fe640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:55.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.242+0000 7f2d7f7fe640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:55.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.242+0000 7f2d7f7fe640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:42:55.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.243+0000 7f2d96d9d640 10 monclient: get_auth_request con 0x7f2d8406e000 auth_method 0 2024-03-29T11:42:55.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.243+0000 7f2d7f7fe640 10 client.4512.objecter ms_handle_connect 0x7f2d8406e000 2024-03-29T11:42:55.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.250+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d981e48a0 con 0x7f2d9813ee90 2024-03-29T11:42:55.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.250+0000 7f2d7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:55.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.250+0000 7f2d7f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:55.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.584+0000 7f2d9d104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:55.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.584+0000 7f2d9d104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:55.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.584+0000 7f2d9d104640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:42:55.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.584+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d980ae670 con 0x7f2d9813ee90 2024-03-29T11:42:55.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.588+0000 7f2d7f7fe640 1 -- 172.21.15.33:0/773652532 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f2d981e48a0 con 0x7f2d9813ee90 2024-03-29T11:42:55.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.588+0000 7f2d7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:55.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.588+0000 7f2d7f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:55.591 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:42:55.591 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:55.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.592+0000 7f2d9d104640 10 librados: watch_flush enter 2024-03-29T11:42:55.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 10 librados: watch_flush exit 2024-03-29T11:42:55.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:42:55.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:42:55.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 >> v1:172.21.15.153:6816/2737028074 conn(0x7f2d8406e000 legacy=0x7f2d840704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:55.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 10 monclient: shutdown 2024-03-29T11:42:55.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:55.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 >> v1:172.21.15.33:6789/0 conn(0x7f2d9813ee90 legacy=0x7f2d98071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:55.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 shutdown_connections 2024-03-29T11:42:55.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.593+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 >> 172.21.15.33:0/773652532 conn(0x7f2d980b35e0 msgr2=0x7f2d980adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:55.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.594+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 shutdown_connections 2024-03-29T11:42:55.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.594+0000 7f2d9d104640 1 -- 172.21.15.33:0/773652532 wait complete. 2024-03-29T11:42:55.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:42:55.594+0000 7f2d9d104640 1 librados: shutdown 2024-03-29T11:42:55.615 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:02.422 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:02.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.561+0000 7fa3069f6640 10 monclient: get_monmap_and_config 2024-03-29T11:43:02.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.561+0000 7fa3069f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:02.565 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.561+0000 7fa3069f6640 10 monclient: monmap: 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:02.562436+0000 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:02.562436+0000 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:02.566 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:02.567 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.563+0000 7fa3069f6640 1 Processor -- start 2024-03-29T11:43:02.567 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.563+0000 7fa3069f6640 1 -- start start 2024-03-29T11:43:02.567 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.563+0000 7fa3069f6640 10 monclient: init 2024-03-29T11:43:02.567 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.564+0000 7fa3069f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:02.567 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.564+0000 7fa3069f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa3000ae250 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa30013f970 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa300142800 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3000afe00 con 0x7fa300142800 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa300142c10 con 0x7fa30013f970 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3000afb20 con 0x7fa3000ae250 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.565+0000 7fa3069f6640 10 monclient(hunting): authenticate will time out at 1338.828781s 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.566+0000 7fa2fffff640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fa3000ae250 0x7fa30013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:42035/0 (socket says 172.21.15.33:42035) 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.566+0000 7fa2fffff640 1 -- 172.21.15.33:0/1754874770 learned_addr learned my addr 172.21.15.33:0/1754874770 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:02.570 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422333809 0 0) 0x7fa3000afb20 con 0x7fa3000ae250 2024-03-29T11:43:02.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:43:02.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e8003520 con 0x7fa3000ae250 2024-03-29T11:43:02.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3669629096 0 0) 0x7fa2e8003520 con 0x7fa3000ae250 2024-03-29T11:43:02.571 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.567+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 >> v1:172.21.15.153:6789/0 conn(0x7fa30013f970 legacy=0x7fa3001420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.570+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 >> v1:172.21.15.33:6789/0 conn(0x7fa300142800 legacy=0x7fa30014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.570+0000 7fa2feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 1 monclient: found mon.noname-b 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3000aab80 con 0x7fa3000ae250 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:02.575 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa2f4001690 con 0x7fa3000ae250 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: dump: 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:02.576 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2f4001920 con 0x7fa3000ae250 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 1 -- 172.21.15.33:0/1754874770 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa2f4001cb0 con 0x7fa3000ae250 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa2feffd640 10 monclient: dump: 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:02.577 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa3069f6640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa3069f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.571+0000 7fa3069f6640 4 set_mon_vals no callback set 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.572+0000 7fa3069f6640 10 monclient: shutdown 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.572+0000 7fa3069f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:02.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.572+0000 7fa3069f6640 1 -- 172.21.15.33:0/1754874770 >> v1:172.21.15.33:6790/0 conn(0x7fa3000ae250 legacy=0x7fa30013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.573+0000 7fa3069f6640 1 -- 172.21.15.33:0/1754874770 shutdown_connections 2024-03-29T11:43:02.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.573+0000 7fa3069f6640 1 -- 172.21.15.33:0/1754874770 >> 172.21.15.33:0/1754874770 conn(0x7fa3000b35e0 msgr2=0x7fa3000b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:02.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.573+0000 7fa3069f6640 1 -- 172.21.15.33:0/1754874770 shutdown_connections 2024-03-29T11:43:02.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.573+0000 7fa3069f6640 1 -- 172.21.15.33:0/1754874770 wait complete. 2024-03-29T11:43:02.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.574+0000 7fa3069f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:02.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.574+0000 7fa3069f6640 10 monclient: monmap: 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:02.582 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.575+0000 7fa3069f6640 1 librados: starting msgr at 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.575+0000 7fa3069f6640 1 librados: starting objecter 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.575+0000 7fa3069f6640 1 Processor -- start 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.576+0000 7fa3069f6640 1 -- start start 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.576+0000 7fa3069f6640 1 librados: setting wanted keys 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.576+0000 7fa3069f6640 1 librados: calling monclient init 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.576+0000 7fa3069f6640 10 monclient: init 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.577+0000 7fa3069f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.577+0000 7fa3069f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.577+0000 7fa3069f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa3000ae250 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:02.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.577+0000 7fa3069f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa30013f970 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:02.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.578+0000 7fa3069f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa300142800 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:02.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.578+0000 7fa3069f6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa300141740 con 0x7fa3000ae250 2024-03-29T11:43:02.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.578+0000 7fa3069f6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3001418d0 con 0x7fa30013f970 2024-03-29T11:43:02.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.578+0000 7fa3069f6640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa300136430 con 0x7fa300142800 2024-03-29T11:43:02.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.578+0000 7fa3069f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:02.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.578+0000 7fa3069f6640 10 monclient(hunting): authenticate will time out at 1338.842123s 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.579+0000 7fa2fffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fa3000ae250 0x7fa30013b870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:34224/0 (socket says 172.21.15.33:34224) 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.579+0000 7fa2fffff640 1 -- 172.21.15.33:0/3047945702 learned_addr learned my addr 172.21.15.33:0/3047945702 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.579+0000 7fa2fcff9640 10 client.?.objecter ms_handle_connect 0x7fa3000ae250 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.579+0000 7fa2fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.580+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375521026 0 0) 0x7fa300141740 con 0x7fa3000ae250 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.580+0000 7fa2fcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.580+0000 7fa2fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.580+0000 7fa2fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.580+0000 7fa2fcff9640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.580+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec004eb0 con 0x7fa3000ae250 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.581+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304345214 0 0) 0x7fa2ec004eb0 con 0x7fa3000ae250 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.581+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa300141740 con 0x7fa3000ae250 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.581+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa2f4001610 con 0x7fa3000ae250 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.581+0000 7fa2fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.581+0000 7fa2fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.581+0000 7fa2fcff9640 10 monclient(hunting): dump: 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.591 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.581+0000 7fa2fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 10 client.?.objecter ms_handle_connect 0x7fa300142800 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193501767 0 0) 0x7fa300136430 con 0x7fa300142800 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:02.592 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.582+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec004eb0 con 0x7fa300142800 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 10 client.?.objecter ms_handle_connect 0x7fa30013f970 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 382401999 0 0) 0x7fa300141740 con 0x7fa3000ae250 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 >> v1:172.21.15.33:6790/0 conn(0x7fa300142800 legacy=0x7fa3001da610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 >> v1:172.21.15.153:6789/0 conn(0x7fa30013f970 legacy=0x7fa300135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 1 monclient: found mon.a 2024-03-29T11:43:02.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3001365c0 con 0x7fa3000ae250 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa2fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 10 monclient: _renew_subs 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30013b240 con 0x7fa3000ae250 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 10 monclient: _renew_subs 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.583+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3001dad20 con 0x7fa3000ae250 2024-03-29T11:43:02.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.584+0000 7fa3069f6640 1 librados: init done 2024-03-29T11:43:02.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.585+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2f4004bf0 con 0x7fa3000ae250 2024-03-29T11:43:02.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.585+0000 7fa2fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.585+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa2f4004f00 con 0x7fa3000ae250 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.585+0000 7fa2fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.585+0000 7fa2fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.585+0000 7fa2fcff9640 10 monclient: dump: 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:02.597 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.585+0000 7fa2fdffb640 4 set_mon_vals no callback set 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.587+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fa2f401aed0 con 0x7fa3000ae250 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.587+0000 7fa3069f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa3069f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa3069f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e0005080 con 0x7fa3000ae250 2024-03-29T11:43:02.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fa2f4084cf0 con 0x7fa3000ae250 2024-03-29T11:43:02.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa2fcff9640 10 client.4518.objecter ms_dispatch 0x7fa300000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:02.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa2fcff9640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:02.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa2fcff9640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:02.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa2fcff9640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:43:02.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.588+0000 7fa2ff7fe640 10 monclient: get_auth_request con 0x7fa2ec06da90 auth_method 0 2024-03-29T11:43:02.599 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.592+0000 7fa2fcff9640 10 client.4518.objecter ms_handle_connect 0x7fa2ec06da90 2024-03-29T11:43:02.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.597+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2f4050400 con 0x7fa3000ae250 2024-03-29T11:43:02.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.597+0000 7fa2fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:02.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.597+0000 7fa2fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:02.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.934+0000 7fa3069f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:02.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.934+0000 7fa3069f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:02.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.934+0000 7fa3069f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:02.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.934+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2e0005330 con 0x7fa3000ae250 2024-03-29T11:43:02.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.935+0000 7fa2fcff9640 1 -- 172.21.15.33:0/3047945702 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fa2f4052500 con 0x7fa3000ae250 2024-03-29T11:43:02.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.935+0000 7fa2fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:02.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.935+0000 7fa2fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:02.939 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:02.939 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:02.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 10 librados: watch_flush enter 2024-03-29T11:43:02.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 10 librados: watch_flush exit 2024-03-29T11:43:02.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:43:02.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:43:02.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 >> v1:172.21.15.153:6816/2737028074 conn(0x7fa2ec06da90 legacy=0x7fa2ec06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 10 monclient: shutdown 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 >> v1:172.21.15.33:6789/0 conn(0x7fa3000ae250 legacy=0x7fa30013b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 shutdown_connections 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.940+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 >> 172.21.15.33:0/3047945702 conn(0x7fa3000b35e0 msgr2=0x7fa30013e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.941+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 shutdown_connections 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.941+0000 7fa3069f6640 1 -- 172.21.15.33:0/3047945702 wait complete. 2024-03-29T11:43:02.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:02.941+0000 7fa3069f6640 1 librados: shutdown 2024-03-29T11:43:02.963 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:09.771 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:09.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.933+0000 7f03288fb640 10 monclient: get_monmap_and_config 2024-03-29T11:43:09.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.933+0000 7f03288fb640 10 monclient: build_initial_monmap 2024-03-29T11:43:09.937 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.933+0000 7f03288fb640 10 monclient: monmap: 2024-03-29T11:43:09.937 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:09.937 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:09.934835+0000 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:09.934835+0000 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:09.938 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:09.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.934+0000 7f03288fb640 1 Processor -- start 2024-03-29T11:43:09.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.934+0000 7f03288fb640 1 -- start start 2024-03-29T11:43:09.939 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.934+0000 7f03288fb640 10 monclient: init 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 10 monclient(hunting): picked mon.noname-c con 0x7f03241422b0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 10 monclient(hunting): picked mon.noname-a con 0x7f0324142690 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 10 monclient(hunting): picked mon.noname-b con 0x7f03240b17d0 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324142c80 con 0x7f0324142690 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240aae40 con 0x7f03241422b0 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032413fbd0 con 0x7f03240b17d0 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.935+0000 7f03288fb640 10 monclient(hunting): authenticate will time out at 1346.200527s 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f03241422b0 0x7f032413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:50602/0 (socket says 172.21.15.33:50602) 2024-03-29T11:43:09.940 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032259c640 1 -- 172.21.15.33:0/3101022093 learned_addr learned my addr 172.21.15.33:0/3101022093 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:09.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206987483 0 0) 0x7f03240aae40 con 0x7f03241422b0 2024-03-29T11:43:09.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032159a640 10 monclient(hunting): _negotiate 2024-03-29T11:43:09.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:09.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:09.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032159a640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:43:09.942 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.936+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003520 con 0x7f03241422b0 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3713213397 0 0) 0x7f0310003520 con 0x7f03241422b0 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 >> v1:172.21.15.33:6790/0 conn(0x7f03240b17d0 legacy=0x7f03240b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 >> v1:172.21.15.33:6789/0 conn(0x7f0324142690 legacy=0x7f03240b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: _finish_hunting 0 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 monclient: found mon.noname-c 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324151940 con 0x7f03241422b0 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: _finish_auth 0 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: _check_auth_tickets 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f031c001690 con 0x7f03241422b0 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: dump: 2024-03-29T11:43:09.943 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f03288fb640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f03288fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f031c0018b0 con 0x7f03241422b0 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 1 -- 172.21.15.33:0/3101022093 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f031c001690 con 0x7f03241422b0 2024-03-29T11:43:09.944 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f032159a640 10 monclient: dump: 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:09.945 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f03288fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:09.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.937+0000 7f03288fb640 4 set_mon_vals no callback set 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.938+0000 7f03288fb640 10 monclient: shutdown 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.938+0000 7f03288fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.938+0000 7f03288fb640 1 -- 172.21.15.33:0/3101022093 >> v1:172.21.15.153:6789/0 conn(0x7f03241422b0 legacy=0x7f032413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.939+0000 7f03288fb640 1 -- 172.21.15.33:0/3101022093 shutdown_connections 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.939+0000 7f03288fb640 1 -- 172.21.15.33:0/3101022093 >> 172.21.15.33:0/3101022093 conn(0x7f0324066a00 msgr2=0x7f0324066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.939+0000 7f03288fb640 1 -- 172.21.15.33:0/3101022093 shutdown_connections 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.939+0000 7f03288fb640 1 -- 172.21.15.33:0/3101022093 wait complete. 2024-03-29T11:43:09.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.940+0000 7f03288fb640 10 monclient: build_initial_monmap 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.940+0000 7f03288fb640 10 monclient: monmap: 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.941+0000 7f03288fb640 1 librados: starting msgr at 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.941+0000 7f03288fb640 1 librados: starting objecter 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.941+0000 7f03288fb640 1 Processor -- start 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.941+0000 7f03288fb640 1 -- start start 2024-03-29T11:43:09.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.941+0000 7f03288fb640 1 librados: setting wanted keys 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.941+0000 7f03288fb640 1 librados: calling monclient init 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.941+0000 7f03288fb640 10 monclient: init 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 10 monclient(hunting): picked mon.noname-b con 0x7f03240b17d0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 10 monclient(hunting): picked mon.noname-c con 0x7f03241422b0 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f032259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f03240b17d0 0x7f03240b5780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:51134/0 (socket says 172.21.15.33:51134) 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 10 monclient(hunting): picked mon.noname-a con 0x7f0324142690 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f032259c640 1 -- 172.21.15.33:0/3062191516 learned_addr learned my addr 172.21.15.33:0/3062191516 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240abac0 con 0x7f0324142690 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240abda0 con 0x7f03240b17d0 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032412e910 con 0x7f03241422b0 2024-03-29T11:43:09.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.942+0000 7f03288fb640 10 monclient(hunting): authenticate will time out at 1346.207625s 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 client.?.objecter ms_handle_connect 0x7f03240b17d0 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392029723 0 0) 0x7f03240abda0 con 0x7f03240b17d0 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304004eb0 con 0x7f03240b17d0 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 client.?.objecter ms_handle_connect 0x7f03241422b0 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3518329818 0 0) 0x7f0304004eb0 con 0x7f03240b17d0 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03240abda0 con 0x7f03240b17d0 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f031c001690 con 0x7f03240b17d0 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:09.953 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): dump: 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:09.954 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:09.955 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:09.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.943+0000 7f031affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:09.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140322121 0 0) 0x7f032412e910 con 0x7f03241422b0 2024-03-29T11:43:09.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:09.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:09.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:09.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:43:09.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304004eb0 con 0x7f03241422b0 2024-03-29T11:43:09.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1032083061 0 0) 0x7f03240abda0 con 0x7f03240b17d0 2024-03-29T11:43:09.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 >> v1:172.21.15.33:6790/0 conn(0x7f03241422b0 legacy=0x7f032414d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 >> v1:172.21.15.33:6789/0 conn(0x7f0324142690 legacy=0x7f03241e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:09.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:09.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 monclient: found mon.b 2024-03-29T11:43:09.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:09.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:09.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03240b5e90 con 0x7f03240b17d0 2024-03-29T11:43:09.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: _finish_auth 0 2024-03-29T11:43:09.958 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:09.958 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:09.958 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:43:09.958 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 10 monclient: _renew_subs 2024-03-29T11:43:09.958 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:09.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032412ee30 con 0x7f03240b17d0 2024-03-29T11:43:09.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:09.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 10 monclient: _renew_subs 2024-03-29T11:43:09.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:09.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0324131650 con 0x7f03240b17d0 2024-03-29T11:43:09.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f03288fb640 1 librados: init done 2024-03-29T11:43:09.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f031c001c00 con 0x7f03240b17d0 2024-03-29T11:43:09.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:09.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f031c004f90 con 0x7f03240b17d0 2024-03-29T11:43:09.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:09.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:09.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f031affd640 10 monclient: dump: 2024-03-29T11:43:09.961 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:09.961 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:09.961 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.961 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:09.961 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:09.962 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:09.962 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:09.962 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:09.962 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:09.962 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:09.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.944+0000 7f0320d99640 4 set_mon_vals no callback set 2024-03-29T11:43:09.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.945+0000 7f03288fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:09.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.945+0000 7f03288fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:09.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.945+0000 7f03288fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:09.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.945+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0308005080 con 0x7f03240b17d0 2024-03-29T11:43:09.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.947+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f031c01af60 con 0x7f03240b17d0 2024-03-29T11:43:09.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.948+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f031c084d60 con 0x7f03240b17d0 2024-03-29T11:43:09.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.948+0000 7f031affd640 10 client.4540.objecter ms_dispatch 0x7f0324000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:09.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.948+0000 7f031affd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:09.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.948+0000 7f031affd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:09.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.948+0000 7f031affd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T11:43:09.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.948+0000 7f0321d9b640 10 monclient: get_auth_request con 0x7f030406ddd0 auth_method 0 2024-03-29T11:43:09.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.948+0000 7f031affd640 10 client.4540.objecter ms_handle_connect 0x7f030406ddd0 2024-03-29T11:43:09.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.953+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f031c050640 con 0x7f03240b17d0 2024-03-29T11:43:09.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.953+0000 7f031affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:09.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:09.953+0000 7f031affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:10.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.390+0000 7f03288fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:10.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.390+0000 7f03288fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:10.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.390+0000 7f03288fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:10.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.390+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0308005330 con 0x7f03240b17d0 2024-03-29T11:43:10.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.391+0000 7f031affd640 1 -- 172.21.15.33:0/3062191516 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f031c052570 con 0x7f03240b17d0 2024-03-29T11:43:10.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.391+0000 7f031affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:10.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.391+0000 7f031affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:10.396 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:10.396 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:10.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 10 librados: watch_flush enter 2024-03-29T11:43:10.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 10 librados: watch_flush exit 2024-03-29T11:43:10.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 10 client.4540.objecter successfully canceled tick 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 >> v1:172.21.15.153:6816/2737028074 conn(0x7f030406ddd0 legacy=0x7f0304070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 10 monclient: shutdown 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 >> v1:172.21.15.153:6789/0 conn(0x7f03240b17d0 legacy=0x7f03240b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 shutdown_connections 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 >> 172.21.15.33:0/3062191516 conn(0x7f0324066a00 msgr2=0x7f03240b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.396+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 shutdown_connections 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.397+0000 7f03288fb640 1 -- 172.21.15.33:0/3062191516 wait complete. 2024-03-29T11:43:10.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:10.397+0000 7f03288fb640 1 librados: shutdown 2024-03-29T11:43:10.416 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:17.224 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.372+0000 7f702e3ad640 10 monclient: get_monmap_and_config 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.372+0000 7f702e3ad640 10 monclient: build_initial_monmap 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.373+0000 7f702e3ad640 10 monclient: monmap: 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:17.374572+0000 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:17.374572+0000 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:17.377 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:17.378 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:17.378 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:17.378 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:17.378 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:17.379 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.374+0000 7f702e3ad640 1 Processor -- start 2024-03-29T11:43:17.379 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.374+0000 7f702e3ad640 1 -- start start 2024-03-29T11:43:17.379 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.374+0000 7f702e3ad640 10 monclient: init 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 10 monclient(hunting): picked mon.noname-a con 0x7f70280b2800 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 10 monclient(hunting): picked mon.noname-c con 0x7f70280a9c00 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 10 monclient(hunting): picked mon.noname-b con 0x7f70280b38c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7028141eb0 con 0x7f70280b2800 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70280b22d0 con 0x7f70280a9c00 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7028142d30 con 0x7f70280b38c0 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:17.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.375+0000 7f702e3ad640 10 monclient(hunting): authenticate will time out at 1353.640512s 2024-03-29T11:43:17.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.377+0000 7f702d3ab640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f70280b2800 0x7f7028142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:35726/0 (socket says 172.21.15.33:35726) 2024-03-29T11:43:17.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.377+0000 7f702d3ab640 1 -- 172.21.15.33:0/1777394557 learned_addr learned my addr 172.21.15.33:0/1777394557 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:17.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.377+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723018298 0 0) 0x7f7028141eb0 con 0x7f70280b2800 2024-03-29T11:43:17.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.378+0000 7f701ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:17.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.378+0000 7f701ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:17.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.378+0000 7f701ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:17.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.378+0000 7f701ffff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:43:17.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.378+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7014003520 con 0x7f70280b2800 2024-03-29T11:43:17.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.379+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1693233228 0 0) 0x7f7014003520 con 0x7f70280b2800 2024-03-29T11:43:17.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.379+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 >> v1:172.21.15.33:6790/0 conn(0x7f70280b38c0 legacy=0x7f702814d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 >> v1:172.21.15.153:6789/0 conn(0x7f70280a9c00 legacy=0x7f702813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 1 monclient: found mon.noname-a 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70280aaf90 con 0x7f70280b2800 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 10 monclient: _finish_auth 0 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f702e3ad640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f702e3ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7024001690 con 0x7f70280b2800 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:17.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f701ffff640 10 monclient: dump: 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:17.386 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.380+0000 7f702e3ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70240018b0 con 0x7f70280b2800 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f701ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f701ffff640 1 -- 172.21.15.33:0/1777394557 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7024001c40 con 0x7f70280b2800 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f701ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f701ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f701ffff640 10 monclient: dump: 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:17.387 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:17.388 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:17.388 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:17.388 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:17.388 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:17.392 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f702e3ad640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:17.392 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.381+0000 7f702e3ad640 4 set_mon_vals no callback set 2024-03-29T11:43:17.392 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.382+0000 7f702e3ad640 10 monclient: shutdown 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.382+0000 7f702e3ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.382+0000 7f702e3ad640 1 -- 172.21.15.33:0/1777394557 >> v1:172.21.15.33:6789/0 conn(0x7f70280b2800 legacy=0x7f7028142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.382+0000 7f702e3ad640 1 -- 172.21.15.33:0/1777394557 shutdown_connections 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.382+0000 7f702e3ad640 1 -- 172.21.15.33:0/1777394557 >> 172.21.15.33:0/1777394557 conn(0x7f7028066a00 msgr2=0x7f7028066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.383+0000 7f702e3ad640 1 -- 172.21.15.33:0/1777394557 shutdown_connections 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.383+0000 7f702e3ad640 1 -- 172.21.15.33:0/1777394557 wait complete. 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.384+0000 7f702e3ad640 10 monclient: build_initial_monmap 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.384+0000 7f702e3ad640 10 monclient: monmap: 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:17.393 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.384+0000 7f702e3ad640 1 librados: starting msgr at 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.384+0000 7f702e3ad640 1 librados: starting objecter 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.384+0000 7f702e3ad640 1 Processor -- start 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 1 -- start start 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 1 librados: setting wanted keys 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 1 librados: calling monclient init 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 10 monclient: init 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 10 monclient(hunting): picked mon.noname-b con 0x7f70280a9c00 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.385+0000 7f702e3ad640 10 monclient(hunting): picked mon.noname-a con 0x7f70280b2800 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.386+0000 7f702e3ad640 10 monclient(hunting): picked mon.noname-c con 0x7f70280b38c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:17.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.386+0000 7f702e3ad640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7028071e30 con 0x7f70280b2800 2024-03-29T11:43:17.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.386+0000 7f702e3ad640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70280ac040 con 0x7f70280a9c00 2024-03-29T11:43:17.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.386+0000 7f702e3ad640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70280ac320 con 0x7f70280b38c0 2024-03-29T11:43:17.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.386+0000 7f702e3ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:17.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.386+0000 7f702e3ad640 10 monclient(hunting): authenticate will time out at 1353.650701s 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.389+0000 7f702d3ab640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f70280a9c00 0x7f70281deba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:52829/0 (socket says 172.21.15.33:52829) 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.389+0000 7f702d3ab640 1 -- 172.21.15.33:0/759414923 learned_addr learned my addr 172.21.15.33:0/759414923 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 client.?.objecter ms_handle_connect 0x7f70280b2800 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 client.?.objecter ms_handle_connect 0x7f70280a9c00 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551674788 0 0) 0x7f7028071e30 con 0x7f70280b2800 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.390+0000 7f701dffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:43:17.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700c004eb0 con 0x7f70280b2800 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 10 client.?.objecter ms_handle_connect 0x7f70280b38c0 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913984095 0 0) 0x7f70280ac040 con 0x7f70280a9c00 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7028071e30 con 0x7f70280a9c00 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275967024 0 0) 0x7f700c004eb0 con 0x7f70280b2800 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.391+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70280ac040 con 0x7f70280b2800 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7020001be0 con 0x7f70280b2800 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 10 monclient(hunting): dump: 2024-03-29T11:43:17.401 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446037995 0 0) 0x7f7028071e30 con 0x7f70280a9c00 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f700c004eb0 con 0x7f70280a9c00 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7024001550 con 0x7f70280a9c00 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:17.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.392+0000 7f701dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 10 monclient(hunting): dump: 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1232517936 0 0) 0x7f700c004eb0 con 0x7f70280a9c00 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 >> v1:172.21.15.33:6790/0 conn(0x7f70280b38c0 legacy=0x7f70281e5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 >> v1:172.21.15.33:6789/0 conn(0x7f70280b2800 legacy=0x7f70281df2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 1 monclient: found mon.b 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702813b1d0 con 0x7f70280a9c00 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.393+0000 7f701dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.394+0000 7f701dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.394+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70240016b0 con 0x7f70280a9c00 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.394+0000 7f701dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.394+0000 7f701f7fe640 4 set_mon_vals no callback set 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 10 monclient: _renew_subs 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702813b480 con 0x7f70280a9c00 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f7024004d90 con 0x7f70280a9c00 2024-03-29T11:43:17.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f701dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f701dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f701dffb640 10 monclient: dump: 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 10 monclient: _renew_subs 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:17.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f702806e4b0 con 0x7f70280a9c00 2024-03-29T11:43:17.406 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.395+0000 7f702e3ad640 1 librados: init done 2024-03-29T11:43:17.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.398+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f702401ad60 con 0x7f70280a9c00 2024-03-29T11:43:17.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f7024084b50 con 0x7f70280a9c00 2024-03-29T11:43:17.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f701dffb640 10 client.4546.objecter ms_dispatch 0x7f7028000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:17.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f702e3ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:17.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f701dffb640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:17.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f701dffb640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:17.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f701dffb640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:43:17.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f702e3ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:17.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f702e3ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:17.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7028071c00 con 0x7f70280a9c00 2024-03-29T11:43:17.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.399+0000 7f702cbaa640 10 monclient: get_auth_request con 0x7f700c06df70 auth_method 0 2024-03-29T11:43:17.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.400+0000 7f701dffb640 10 client.4546.objecter ms_handle_connect 0x7f700c06df70 2024-03-29T11:43:17.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.406+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7028071c00 con 0x7f70280a9c00 2024-03-29T11:43:17.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.406+0000 7f701dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:17.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.406+0000 7f701dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:17.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.860+0000 7f702e3ad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:17.864 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.860+0000 7f702e3ad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:17.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.860+0000 7f702e3ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:17.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.860+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7028071c00 con 0x7f70280a9c00 2024-03-29T11:43:17.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.861+0000 7f701dffb640 1 -- 172.21.15.33:0/759414923 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f7028071c00 con 0x7f70280a9c00 2024-03-29T11:43:17.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.861+0000 7f701dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:17.865 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.861+0000 7f701dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:17.866 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:17.866 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 10 librados: watch_flush enter 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 10 librados: watch_flush exit 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 >> v1:172.21.15.153:6816/2737028074 conn(0x7f700c06df70 legacy=0x7f700c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 10 monclient: shutdown 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 >> v1:172.21.15.153:6789/0 conn(0x7f70280a9c00 legacy=0x7f70281deba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:17.870 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.865+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 shutdown_connections 2024-03-29T11:43:17.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.866+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 >> 172.21.15.33:0/759414923 conn(0x7f7028066a00 msgr2=0x7f70281451f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:17.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.866+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 shutdown_connections 2024-03-29T11:43:17.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.866+0000 7f702e3ad640 1 -- 172.21.15.33:0/759414923 wait complete. 2024-03-29T11:43:17.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:17.866+0000 7f702e3ad640 1 librados: shutdown 2024-03-29T11:43:17.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:24.699 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:24.848 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.844+0000 7fd49f484640 10 monclient: get_monmap_and_config 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.844+0000 7fd49f484640 10 monclient: build_initial_monmap 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.844+0000 7fd49f484640 10 monclient: monmap: 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:24.846177+0000 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:24.846177+0000 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:24.849 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:24.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.845+0000 7fd49f484640 1 Processor -- start 2024-03-29T11:43:24.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.845+0000 7fd49f484640 1 -- start start 2024-03-29T11:43:24.850 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.845+0000 7fd49f484640 10 monclient: init 2024-03-29T11:43:24.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.846+0000 7fd49f484640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:24.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.847+0000 7fd49f484640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:24.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.847+0000 7fd49f484640 10 monclient(hunting): picked mon.noname-b con 0x7fd4980b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:24.851 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.847+0000 7fd49f484640 10 monclient(hunting): picked mon.noname-a con 0x7fd49814a700 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:24.853 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49f484640 10 monclient(hunting): picked mon.noname-c con 0x7fd4980ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:24.853 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49f484640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980aba70 con 0x7fd49814a700 2024-03-29T11:43:24.853 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49f484640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980aecb0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.853 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49f484640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980af150 con 0x7fd4980b0770 2024-03-29T11:43:24.853 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49f484640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:24.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49f484640 10 monclient(hunting): authenticate will time out at 1361.114379s 2024-03-29T11:43:24.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49d220640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fd4980b0770 0x7fd498149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:51468/0 (socket says 172.21.15.33:51468) 2024-03-29T11:43:24.854 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.849+0000 7fd49d220640 1 -- 172.21.15.33:0/3445713532 learned_addr learned my addr 172.21.15.33:0/3445713532 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:24.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062455426 0 0) 0x7fd4980aba70 con 0x7fd49814a700 2024-03-29T11:43:24.855 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd484003520 con 0x7fd49814a700 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272506192 0 0) 0x7fd4980af150 con 0x7fd4980b0770 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.856 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.850+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4980aba70 con 0x7fd4980b0770 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558195978 0 0) 0x7fd484003520 con 0x7fd49814a700 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 >> v1:172.21.15.33:6790/0 conn(0x7fd4980b0770 legacy=0x7fd498149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 >> v1:172.21.15.153:6789/0 conn(0x7fd4980ac5a0 legacy=0x7fd498151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 monclient: found mon.noname-a 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980ac9d0 con 0x7fd49814a700 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: _finish_auth 0 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:24.857 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:24.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd494001690 con 0x7fd49814a700 2024-03-29T11:43:24.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:24.858 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: dump: 2024-03-29T11:43:24.858 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:24.858 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd49f484640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd49f484640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4940017f0 con 0x7fd49814a700 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 1 -- 172.21.15.33:0/3445713532 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd494001c00 con 0x7fd49814a700 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd49f484640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:24.859 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd49f484640 4 set_mon_vals no callback set 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.851+0000 7fd48ffff640 10 monclient: dump: 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:24.860 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:24.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.852+0000 7fd49f484640 10 monclient: shutdown 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.852+0000 7fd49f484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.852+0000 7fd49f484640 1 -- 172.21.15.33:0/3445713532 >> v1:172.21.15.33:6789/0 conn(0x7fd49814a700 legacy=0x7fd4980abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.852+0000 7fd49f484640 1 -- 172.21.15.33:0/3445713532 shutdown_connections 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.852+0000 7fd49f484640 1 -- 172.21.15.33:0/3445713532 >> 172.21.15.33:0/3445713532 conn(0x7fd49806b660 msgr2=0x7fd49806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.853+0000 7fd49f484640 1 -- 172.21.15.33:0/3445713532 shutdown_connections 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.854+0000 7fd49f484640 1 -- 172.21.15.33:0/3445713532 wait complete. 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.855+0000 7fd49f484640 10 monclient: build_initial_monmap 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.855+0000 7fd49f484640 10 monclient: monmap: 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:24.875 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.856+0000 7fd49f484640 1 librados: starting msgr at 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.856+0000 7fd49f484640 1 librados: starting objecter 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.856+0000 7fd49f484640 1 Processor -- start 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.856+0000 7fd49f484640 1 -- start start 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.856+0000 7fd49f484640 1 librados: setting wanted keys 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.856+0000 7fd49f484640 1 librados: calling monclient init 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.856+0000 7fd49f484640 10 monclient: init 2024-03-29T11:43:24.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 10 monclient(hunting): picked mon.noname-b con 0x7fd4980ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 10 monclient(hunting): picked mon.noname-c con 0x7fd4980b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 10 monclient(hunting): picked mon.noname-a con 0x7fd498125690 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd498129030 con 0x7fd498125690 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd498127ff0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd498128c00 con 0x7fd4980b0770 2024-03-29T11:43:24.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.857+0000 7fd49f484640 10 monclient(hunting): authenticate will time out at 1361.122334s 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.859+0000 7fd49ca1f640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fd4980b0770 0x7fd4981dbf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:54850/0 (socket says 172.21.15.33:54850) 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.859+0000 7fd49ca1f640 1 -- 172.21.15.33:0/3159768936 learned_addr learned my addr 172.21.15.33:0/3159768936 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.859+0000 7fd48dffb640 10 client.?.objecter ms_handle_connect 0x7fd4980b0770 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.859+0000 7fd48dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 client.?.objecter ms_handle_connect 0x7fd4980ac5a0 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477695263 0 0) 0x7fd498127ff0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c004eb0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650561120 0 0) 0x7fd498128c00 con 0x7fd4980b0770 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.863+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498127ff0 con 0x7fd4980b0770 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 10 client.?.objecter ms_handle_connect 0x7fd498125690 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918755616 0 0) 0x7fd47c004eb0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd498128c00 con 0x7fd4980ac5a0 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd488001850 con 0x7fd4980ac5a0 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 10 monclient(hunting): dump: 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:24.879 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.864+0000 7fd48dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2603496139 0 0) 0x7fd498127ff0 con 0x7fd4980b0770 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd47c004eb0 con 0x7fd4980b0770 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594217808 0 0) 0x7fd498129030 con 0x7fd498125690 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:24.880 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498127ff0 con 0x7fd498125690 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1267411695 0 0) 0x7fd498128c00 con 0x7fd4980ac5a0 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 >> v1:172.21.15.33:6790/0 conn(0x7fd4980b0770 legacy=0x7fd4981dbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 >> v1:172.21.15.33:6789/0 conn(0x7fd498125690 legacy=0x7fd498146f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 1 monclient: found mon.b 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.865+0000 7fd48dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4981dc6a0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd48dffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd48dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd48dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 10 monclient: _renew_subs 2024-03-29T11:43:24.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4981293c0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 10 monclient: _renew_subs 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd498147850 con 0x7fd4980ac5a0 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.866+0000 7fd49f484640 1 librados: init done 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.867+0000 7fd49f484640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.867+0000 7fd49f484640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.867+0000 7fd49f484640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.867+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd498147af0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.868+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd488001d60 con 0x7fd4980ac5a0 2024-03-29T11:43:24.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.868+0000 7fd48dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:24.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.868+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fd488004980 con 0x7fd4980ac5a0 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.868+0000 7fd48dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.868+0000 7fd48dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.868+0000 7fd48dffb640 10 monclient: dump: 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:24.884 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:24.885 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:24.885 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:24.885 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.869+0000 7fd48f7fe640 4 set_mon_vals no callback set 2024-03-29T11:43:24.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.872+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fd48801a950 con 0x7fd4980ac5a0 2024-03-29T11:43:24.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.873+0000 7fd49ca1f640 10 monclient: get_auth_request con 0x7fd47c06dfa0 auth_method 0 2024-03-29T11:43:24.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.873+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fd4880848b0 con 0x7fd4980ac5a0 2024-03-29T11:43:24.886 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.874+0000 7fd48dffb640 10 client.4552.objecter ms_dispatch 0x7fd498000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:24.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.874+0000 7fd48dffb640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:24.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.874+0000 7fd48dffb640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:24.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.874+0000 7fd48dffb640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-29T11:43:24.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.874+0000 7fd48dffb640 10 client.4552.objecter ms_handle_connect 0x7fd47c06dfa0 2024-03-29T11:43:24.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.877+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd488050190 con 0x7fd4980ac5a0 2024-03-29T11:43:24.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.877+0000 7fd48dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:24.887 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:24.877+0000 7fd48dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:25.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.198+0000 7fd49f484640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:25.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.198+0000 7fd49f484640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:25.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.198+0000 7fd49f484640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:25.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.198+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4980ab560 con 0x7fd4980ac5a0 2024-03-29T11:43:25.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.201+0000 7fd48dffb640 1 -- 172.21.15.33:0/3159768936 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fd4880520c0 con 0x7fd4980ac5a0 2024-03-29T11:43:25.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.201+0000 7fd48dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:25.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.201+0000 7fd48dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:25.206 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:25.206 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:25.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.205+0000 7fd49f484640 10 librados: watch_flush enter 2024-03-29T11:43:25.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.205+0000 7fd49f484640 10 librados: watch_flush exit 2024-03-29T11:43:25.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.205+0000 7fd49f484640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-29T11:43:25.209 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.205+0000 7fd49f484640 10 client.4552.objecter successfully canceled tick 2024-03-29T11:43:25.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 >> v1:172.21.15.153:6816/2737028074 conn(0x7fd47c06dfa0 legacy=0x7fd47c070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:25.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 10 monclient: shutdown 2024-03-29T11:43:25.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:25.211 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 >> v1:172.21.15.153:6789/0 conn(0x7fd4980ac5a0 legacy=0x7fd4981284f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:25.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 shutdown_connections 2024-03-29T11:43:25.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 >> 172.21.15.33:0/3159768936 conn(0x7fd49806b660 msgr2=0x7fd4980b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:25.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 shutdown_connections 2024-03-29T11:43:25.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.206+0000 7fd49f484640 1 -- 172.21.15.33:0/3159768936 wait complete. 2024-03-29T11:43:25.212 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:25.207+0000 7fd49f484640 1 librados: shutdown 2024-03-29T11:43:25.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:32.035 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.181+0000 7f26abac0640 10 monclient: get_monmap_and_config 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.181+0000 7f26abac0640 10 monclient: build_initial_monmap 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.181+0000 7f26abac0640 10 monclient: monmap: 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:32.181729+0000 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:32.181729+0000 2024-03-29T11:43:32.184 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.182+0000 7f26abac0640 1 Processor -- start 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.182+0000 7f26abac0640 1 -- start start 2024-03-29T11:43:32.185 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.182+0000 7f26abac0640 10 monclient: init 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 10 monclient(hunting): picked mon.noname-c con 0x7f26a40b4790 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 10 monclient(hunting): picked mon.noname-b con 0x7f26a40aea90 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 10 monclient(hunting): picked mon.noname-a con 0x7f26a40b39f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a40b0c60 con 0x7f26a40b39f0 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a40af020 con 0x7f26a40b4790 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a413fbd0 con 0x7f26a40aea90 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:32.187 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.183+0000 7f26abac0640 10 monclient(hunting): authenticate will time out at 1368.447454s 2024-03-29T11:43:32.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.185+0000 7f26a985c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f26a40b4790 0x7f26a40b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:33288/0 (socket says 172.21.15.33:33288) 2024-03-29T11:43:32.188 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.185+0000 7f26a985c640 1 -- 172.21.15.33:0/3923283394 learned_addr learned my addr 172.21.15.33:0/3923283394 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:32.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.186+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703084663 0 0) 0x7f26a40af020 con 0x7f26a40b4790 2024-03-29T11:43:32.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.186+0000 7f26a885a640 10 monclient(hunting): _negotiate 2024-03-29T11:43:32.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.186+0000 7f26a885a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:32.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.186+0000 7f26a885a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:32.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.186+0000 7f26a885a640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:43:32.189 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.186+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2690003520 con 0x7f26a40b4790 2024-03-29T11:43:32.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 14713334 0 0) 0x7f2690003520 con 0x7f26a40b4790 2024-03-29T11:43:32.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 >> v1:172.21.15.33:6790/0 conn(0x7f26a40aea90 legacy=0x7f26a40af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:32.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 >> v1:172.21.15.33:6789/0 conn(0x7f26a40b39f0 legacy=0x7f26a413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:32.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 10 monclient: _finish_hunting 0 2024-03-29T11:43:32.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 1 monclient: found mon.noname-c 2024-03-29T11:43:32.190 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a413fd60 con 0x7f26a40b4790 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 10 monclient: _finish_auth 0 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 10 monclient: _check_auth_tickets 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f26a0001690 con 0x7f26a40b4790 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26a885a640 10 monclient: dump: 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:32.191 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:32.192 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:32.192 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:32.192 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:32.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26abac0640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:43:32.192 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.187+0000 7f26abac0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:32.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26a0001860 con 0x7f26a40b4790 2024-03-29T11:43:32.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26a885a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:32.193 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26a885a640 1 -- 172.21.15.33:0/3923283394 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f26a0001cb0 con 0x7f26a40b4790 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26a885a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26a885a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26a885a640 10 monclient: dump: 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26abac0640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:32.194 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.188+0000 7f26abac0640 4 set_mon_vals no callback set 2024-03-29T11:43:32.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.189+0000 7f26abac0640 10 monclient: shutdown 2024-03-29T11:43:32.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.189+0000 7f26abac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:32.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.189+0000 7f26abac0640 1 -- 172.21.15.33:0/3923283394 >> v1:172.21.15.153:6789/0 conn(0x7f26a40b4790 legacy=0x7f26a40b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:32.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.191+0000 7f26abac0640 1 -- 172.21.15.33:0/3923283394 shutdown_connections 2024-03-29T11:43:32.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.191+0000 7f26abac0640 1 -- 172.21.15.33:0/3923283394 >> 172.21.15.33:0/3923283394 conn(0x7f26a4066a00 msgr2=0x7f26a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:32.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.192+0000 7f26abac0640 1 -- 172.21.15.33:0/3923283394 shutdown_connections 2024-03-29T11:43:32.195 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.192+0000 7f26abac0640 1 -- 172.21.15.33:0/3923283394 wait complete. 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 10 monclient: build_initial_monmap 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 10 monclient: monmap: 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:32.196 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 1 librados: starting msgr at 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 1 librados: starting objecter 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 1 Processor -- start 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 1 -- start start 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 1 librados: setting wanted keys 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 1 librados: calling monclient init 2024-03-29T11:43:32.197 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.193+0000 7f26abac0640 10 monclient: init 2024-03-29T11:43:32.198 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.194+0000 7f26abac0640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.194+0000 7f26abac0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 10 monclient(hunting): picked mon.noname-a con 0x7f26a40b39f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 10 monclient(hunting): picked mon.noname-b con 0x7f26a40b4790 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 10 monclient(hunting): picked mon.noname-c con 0x7f26a40701d0 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a4071050 con 0x7f26a40b39f0 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a406f5b0 con 0x7f26a40b4790 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a41e4d40 con 0x7f26a40701d0 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:32.199 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26abac0640 10 monclient(hunting): authenticate will time out at 1368.459900s 2024-03-29T11:43:32.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26a905b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f26a40b4790 0x7f26a406eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:34074/0 (socket says 172.21.15.33:34074) 2024-03-29T11:43:32.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f26a905b640 1 -- 172.21.15.33:0/1543992488 learned_addr learned my addr 172.21.15.33:0/1543992488 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:32.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.196+0000 7f269a7fc640 10 client.?.objecter ms_handle_connect 0x7f26a40b4790 2024-03-29T11:43:32.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:32.200 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800308172 0 0) 0x7f26a406f5b0 con 0x7f26a40b4790 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2688004eb0 con 0x7f26a40b4790 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3321607399 0 0) 0x7f2688004eb0 con 0x7f26a40b4790 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26a406f5b0 con 0x7f26a40b4790 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2694001960 con 0x7f26a40b4790 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.201 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:32.202 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:32.202 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:32.202 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:32.202 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:32.202 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:32.202 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.197+0000 7f269a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.198+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2582819025 0 0) 0x7f26a406f5b0 con 0x7f26a40b4790 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.198+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 >> v1:172.21.15.33:6790/0 conn(0x7f26a40701d0 legacy=0x7f26a41e4630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 >> v1:172.21.15.33:6789/0 conn(0x7f26a40b39f0 legacy=0x7f26a41e3f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 1 monclient: found mon.b 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a41309f0 con 0x7f26a40b4790 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f269a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 10 monclient: _renew_subs 2024-03-29T11:43:32.203 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a4130cd0 con 0x7f26a40b4790 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 10 monclient: _renew_subs 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a41e7d80 con 0x7f26a40b4790 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.199+0000 7f26abac0640 1 librados: init done 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2694001c80 con 0x7f26a40b4790 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f269a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f269bfff640 4 set_mon_vals no callback set 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2694003a40 con 0x7f26a40b4790 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f269a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:32.204 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f269a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f269a7fc640 10 monclient: dump: 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:32.205 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.201+0000 7f26abac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.202+0000 7f26abac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.202+0000 7f26abac0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.202+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c005080 con 0x7f26a40b4790 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.202+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f2694003ba0 con 0x7f26a40b4790 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.203+0000 7f26a985c640 10 monclient: get_auth_request con 0x7f268806dc70 auth_method 0 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.203+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f2694084b40 con 0x7f26a40b4790 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.203+0000 7f269a7fc640 10 client.4558.objecter ms_dispatch 0x7f26a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.203+0000 7f269a7fc640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.203+0000 7f269a7fc640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.204+0000 7f269a7fc640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-29T11:43:32.207 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.204+0000 7f269a7fc640 10 client.4558.objecter ms_handle_connect 0x7f268806dc70 2024-03-29T11:43:32.214 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.211+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2694050420 con 0x7f26a40b4790 2024-03-29T11:43:32.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.211+0000 7f269a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:32.215 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.211+0000 7f269a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:32.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.542+0000 7f26abac0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:32.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.542+0000 7f26abac0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:32.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.542+0000 7f26abac0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:32.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.542+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f268c005330 con 0x7f26a40b4790 2024-03-29T11:43:32.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.543+0000 7f269a7fc640 1 -- 172.21.15.33:0/1543992488 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f2694052350 con 0x7f26a40b4790 2024-03-29T11:43:32.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.543+0000 7f269a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:32.546 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.543+0000 7f269a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:32.547 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:32.548 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:32.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 10 librados: watch_flush enter 2024-03-29T11:43:32.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 10 librados: watch_flush exit 2024-03-29T11:43:32.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-29T11:43:32.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 10 client.4558.objecter successfully canceled tick 2024-03-29T11:43:32.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 >> v1:172.21.15.153:6816/2737028074 conn(0x7f268806dc70 legacy=0x7f2688070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:32.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 10 monclient: shutdown 2024-03-29T11:43:32.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:32.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 >> v1:172.21.15.153:6789/0 conn(0x7f26a40b4790 legacy=0x7f26a406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:32.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 shutdown_connections 2024-03-29T11:43:32.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.548+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 >> 172.21.15.33:0/1543992488 conn(0x7f26a4066a00 msgr2=0x7f26a40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:32.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.550+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 shutdown_connections 2024-03-29T11:43:32.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.551+0000 7f26abac0640 1 -- 172.21.15.33:0/1543992488 wait complete. 2024-03-29T11:43:32.554 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:32.551+0000 7f26abac0640 1 librados: shutdown 2024-03-29T11:43:32.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:39.377 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:39.532 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.525+0000 7f4ec962a640 10 monclient: get_monmap_and_config 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.525+0000 7f4ec962a640 10 monclient: build_initial_monmap 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.525+0000 7f4ec962a640 10 monclient: monmap: 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:39.526067+0000 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:39.526067+0000 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:39.533 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:39.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.526+0000 7f4ec962a640 1 Processor -- start 2024-03-29T11:43:39.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.526+0000 7f4ec962a640 1 -- start start 2024-03-29T11:43:39.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.526+0000 7f4ec962a640 10 monclient: init 2024-03-29T11:43:39.534 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:39.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:39.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec40a9750 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:39.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec40ad020 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:39.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec40ad4d0 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:39.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec41533f0 con 0x7f4ec40ad4d0 2024-03-29T11:43:39.535 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40abf30 con 0x7f4ec40ad020 2024-03-29T11:43:39.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40ac210 con 0x7f4ec40a9750 2024-03-29T11:43:39.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:39.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.527+0000 7f4ec962a640 10 monclient(hunting): authenticate will time out at 1375.790999s 2024-03-29T11:43:39.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.529+0000 7f4ec27fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f4ec40ad020 0x7f4ec41525d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:46194/0 (socket says 172.21.15.33:46194) 2024-03-29T11:43:39.536 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.529+0000 7f4ec27fc640 1 -- 172.21.15.33:0/2221642203 learned_addr learned my addr 172.21.15.33:0/2221642203 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:39.538 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162120537 0 0) 0x7f4ec40ac210 con 0x7f4ec40a9750 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eac003520 con 0x7f4ec40a9750 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925436803 0 0) 0x7f4ec40abf30 con 0x7f4ec40ad020 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec40ac210 con 0x7f4ec40ad020 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698108647 0 0) 0x7f4eac003520 con 0x7f4ec40a9750 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 >> v1:172.21.15.153:6789/0 conn(0x7f4ec40ad020 legacy=0x7f4ec41525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:39.539 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.530+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 >> v1:172.21.15.33:6789/0 conn(0x7f4ec40ad4d0 legacy=0x7f4ec4152ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40ac4f0 con 0x7f4ec40a9750 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:39.540 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4eb4001690 con 0x7f4ec40a9750 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec1ffb640 10 monclient: dump: 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:39.541 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec962a640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.531+0000 7f4ec962a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4eb4001870 con 0x7f4ec40a9750 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec962a640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec962a640 4 set_mon_vals no callback set 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec962a640 10 monclient: shutdown 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec962a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec962a640 1 -- 172.21.15.33:0/2221642203 >> v1:172.21.15.33:6790/0 conn(0x7f4ec40a9750 legacy=0x7f4ec40ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec1ffb640 1 -- 172.21.15.33:0/2221642203 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4eb4001c40 con 0x7f4ec40a9750 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.532+0000 7f4ec1ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.533+0000 7f4ec962a640 1 -- 172.21.15.33:0/2221642203 shutdown_connections 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.533+0000 7f4ec962a640 1 -- 172.21.15.33:0/2221642203 >> 172.21.15.33:0/2221642203 conn(0x7f4ec40b3a00 msgr2=0x7f4ec40b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.534+0000 7f4ec962a640 1 -- 172.21.15.33:0/2221642203 shutdown_connections 2024-03-29T11:43:39.542 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.534+0000 7f4ec962a640 1 -- 172.21.15.33:0/2221642203 wait complete. 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.535+0000 7f4ec962a640 10 monclient: build_initial_monmap 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.535+0000 7f4ec962a640 10 monclient: monmap: 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:39.543 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:39.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.536+0000 7f4ec962a640 1 librados: starting msgr at 2024-03-29T11:43:39.544 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.536+0000 7f4ec962a640 1 librados: starting objecter 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.536+0000 7f4ec962a640 1 Processor -- start 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.536+0000 7f4ec962a640 1 -- start start 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.536+0000 7f4ec962a640 1 librados: setting wanted keys 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.536+0000 7f4ec962a640 1 librados: calling monclient init 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.536+0000 7f4ec962a640 10 monclient: init 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.537+0000 7f4ec962a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.537+0000 7f4ec962a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.537+0000 7f4ec962a640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec40a9750 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.537+0000 7f4ec962a640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec40ad020 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec27fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f4ec40ad020 0x7f4ec408a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:46482/0 (socket says 172.21.15.33:46482) 2024-03-29T11:43:39.547 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec27fc640 1 -- 172.21.15.33:0/2718324710 learned_addr learned my addr 172.21.15.33:0/2718324710 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec962a640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec4071f30 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec408af90 con 0x7f4ec4071f30 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec414e130 con 0x7f4ec40ad020 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec406f030 con 0x7f4ec40a9750 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec962a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4ec962a640 10 monclient(hunting): authenticate will time out at 1375.803253s 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4eab7fe640 10 client.?.objecter ms_handle_connect 0x7f4ec4071f30 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4eab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593164284 0 0) 0x7f4ec408af90 con 0x7f4ec4071f30 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4eab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4eab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.539+0000 7f4eab7fe640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.540+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0004eb0 con 0x7f4ec4071f30 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.541+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 721156390 0 0) 0x7f4eb0004eb0 con 0x7f4ec4071f30 2024-03-29T11:43:39.548 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.541+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ec408af90 con 0x7f4ec4071f30 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 client.?.objecter ms_handle_connect 0x7f4ec40ad020 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4eb8001750 con 0x7f4ec4071f30 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): dump: 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:39.550 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041533239 0 0) 0x7f4ec414e130 con 0x7f4ec40ad020 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:43:39.551 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0004eb0 con 0x7f4ec40ad020 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3150923005 0 0) 0x7f4ec408af90 con 0x7f4ec4071f30 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.542+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 >> v1:172.21.15.33:6790/0 conn(0x7f4ec40a9750 legacy=0x7f4ec41e40d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 >> v1:172.21.15.153:6789/0 conn(0x7f4ec40ad020 legacy=0x7f4ec408a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 1 monclient: found mon.a 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec406f1c0 con 0x7f4ec4071f30 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4eab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 10 monclient: _renew_subs 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:39.552 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec41e47e0 con 0x7f4ec4071f30 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 10 monclient: _renew_subs 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec414d320 con 0x7f4ec4071f30 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.543+0000 7f4ec962a640 1 librados: init done 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4ec962a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4ec962a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4ec962a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4eb80018d0 con 0x7f4ec4071f30 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec41e3d70 con 0x7f4ec4071f30 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4eab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4ec17fa640 4 set_mon_vals no callback set 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4eb8003790 con 0x7f4ec4071f30 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4eab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4eab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:39.553 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.544+0000 7f4eab7fe640 10 monclient: dump: 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:39.554 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.547+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f4eb80038f0 con 0x7f4ec4071f30 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.547+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f4eb8083750 con 0x7f4ec4071f30 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.547+0000 7f4eab7fe640 10 client.4536.objecter ms_dispatch 0x7f4ec4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.547+0000 7f4eab7fe640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.547+0000 7f4eab7fe640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.547+0000 7f4eab7fe640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.549+0000 7f4ec2ffd640 10 monclient: get_auth_request con 0x7f4eb006dce0 auth_method 0 2024-03-29T11:43:39.555 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.550+0000 7f4eab7fe640 10 client.4536.objecter ms_handle_connect 0x7f4eb006dce0 2024-03-29T11:43:39.558 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.554+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4eb804ee60 con 0x7f4ec4071f30 2024-03-29T11:43:39.558 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.554+0000 7f4eab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:39.558 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.554+0000 7f4eab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.886+0000 7f4ec962a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.886+0000 7f4ec962a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:39.889 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.886+0000 7f4ec962a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:43:39.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.886+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ec41e3f00 con 0x7f4ec4071f30 2024-03-29T11:43:39.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.887+0000 7f4eab7fe640 1 -- 172.21.15.33:0/2718324710 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f4eb8050f60 con 0x7f4ec4071f30 2024-03-29T11:43:39.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.887+0000 7f4eab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:39.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.887+0000 7f4eab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:39.890 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:39.891 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.890+0000 7f4ec962a640 10 librados: watch_flush enter 2024-03-29T11:43:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.890+0000 7f4ec962a640 10 librados: watch_flush exit 2024-03-29T11:43:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.890+0000 7f4ec962a640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T11:43:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.890+0000 7f4ec962a640 10 client.4536.objecter successfully canceled tick 2024-03-29T11:43:39.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.891+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 >> v1:172.21.15.153:6816/2737028074 conn(0x7f4eb006dce0 legacy=0x7f4eb00701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.891+0000 7f4ec962a640 10 monclient: shutdown 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.891+0000 7f4ec962a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.891+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 >> v1:172.21.15.33:6789/0 conn(0x7f4ec4071f30 legacy=0x7f4ec408a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.891+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 shutdown_connections 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.891+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 >> 172.21.15.33:0/2718324710 conn(0x7f4ec40b3a00 msgr2=0x7f4ec40b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.892+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 shutdown_connections 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.892+0000 7f4ec962a640 1 -- 172.21.15.33:0/2718324710 wait complete. 2024-03-29T11:43:39.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:39.892+0000 7f4ec962a640 1 librados: shutdown 2024-03-29T11:43:39.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:46.722 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:46.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.864+0000 7fa1ab815640 10 monclient: get_monmap_and_config 2024-03-29T11:43:46.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.864+0000 7fa1ab815640 10 monclient: build_initial_monmap 2024-03-29T11:43:46.871 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.864+0000 7fa1ab815640 10 monclient: monmap: 2024-03-29T11:43:46.871 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:46.871 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:46.871 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:46.864960+0000 2024-03-29T11:43:46.871 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:46.864960+0000 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.865+0000 7fa1ab815640 1 Processor -- start 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.865+0000 7fa1ab815640 1 -- start start 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.865+0000 7fa1ab815640 10 monclient: init 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.866+0000 7fa1ab815640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.866+0000 7fa1ab815640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.866+0000 7fa1ab815640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a40ad6c0 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:46.872 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.866+0000 7fa1ab815640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a40af6c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:46.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1ab815640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a414fdc0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:46.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1ab815640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a40a94e0 con 0x7fa1a40ad6c0 2024-03-29T11:43:46.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1ab815640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a40af3a0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.874 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1ab815640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a40afb60 con 0x7fa1a40af6c0 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1ab815640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1ab815640 10 monclient(hunting): authenticate will time out at 1383.131997s 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1a9db2640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa1a414fdc0 0x7fa1a415a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:59904/0 (socket says 172.21.15.33:59904) 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.868+0000 7fa1a9db2640 1 -- 172.21.15.33:0/3194411444 learned_addr learned my addr 172.21.15.33:0/3194411444 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602475606 0 0) 0x7fa1a40af3a0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa190003520 con 0x7fa1a414fdc0 2024-03-29T11:43:46.875 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 489588137 0 0) 0x7fa190003520 con 0x7fa1a414fdc0 2024-03-29T11:43:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.869+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 >> v1:172.21.15.33:6790/0 conn(0x7fa1a40af6c0 legacy=0x7fa1a4152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 >> v1:172.21.15.33:6789/0 conn(0x7fa1a40ad6c0 legacy=0x7fa1a414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 1 monclient: found mon.noname-c 2024-03-29T11:43:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:43:46.876 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a40aab80 con 0x7fa1a414fdc0 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa19c0019f0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa19bfff640 10 monclient: dump: 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:46.877 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa1ab815640 5 monclient: authenticate success, global_id 4567 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.872+0000 7fa1ab815640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa19c001c50 con 0x7fa1a414fdc0 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa19bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa19bfff640 1 -- 172.21.15.33:0/3194411444 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa19c0020a0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa19bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa19bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa19bfff640 10 monclient: dump: 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:46.878 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:46.879 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:46.879 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:46.879 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:46.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa1ab815640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:46.879 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.873+0000 7fa1ab815640 4 set_mon_vals no callback set 2024-03-29T11:43:46.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.874+0000 7fa1ab815640 10 monclient: shutdown 2024-03-29T11:43:46.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.874+0000 7fa1ab815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:46.881 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.874+0000 7fa1ab815640 1 -- 172.21.15.33:0/3194411444 >> v1:172.21.15.153:6789/0 conn(0x7fa1a414fdc0 legacy=0x7fa1a415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.874+0000 7fa1ab815640 1 -- 172.21.15.33:0/3194411444 shutdown_connections 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.874+0000 7fa1ab815640 1 -- 172.21.15.33:0/3194411444 >> 172.21.15.33:0/3194411444 conn(0x7fa1a40b35e0 msgr2=0x7fa1a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.875+0000 7fa1ab815640 1 -- 172.21.15.33:0/3194411444 shutdown_connections 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.875+0000 7fa1ab815640 1 -- 172.21.15.33:0/3194411444 wait complete. 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.876+0000 7fa1ab815640 10 monclient: build_initial_monmap 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.876+0000 7fa1ab815640 10 monclient: monmap: 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:46.882 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.876+0000 7fa1ab815640 1 librados: starting msgr at 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.876+0000 7fa1ab815640 1 librados: starting objecter 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.876+0000 7fa1ab815640 1 Processor -- start 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 1 -- start start 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 1 librados: setting wanted keys 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 1 librados: calling monclient init 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient: init 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a40ad6c0 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a40af6c0 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a414fdc0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a4071a40 con 0x7fa1a40af6c0 2024-03-29T11:43:46.883 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a4162820 con 0x7fa1a414fdc0 2024-03-29T11:43:46.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a4143fe0 con 0x7fa1a40ad6c0 2024-03-29T11:43:46.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:46.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.877+0000 7fa1ab815640 10 monclient(hunting): authenticate will time out at 1383.141629s 2024-03-29T11:43:46.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.878+0000 7fa1a9db2640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa1a414fdc0 0x7fa1a41f74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:60304/0 (socket says 172.21.15.33:60304) 2024-03-29T11:43:46.884 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.878+0000 7fa1a9db2640 1 -- 172.21.15.33:0/1789904874 learned_addr learned my addr 172.21.15.33:0/1789904874 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 10 client.?.objecter ms_handle_connect 0x7fa1a414fdc0 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147278472 0 0) 0x7fa1a4162820 con 0x7fa1a414fdc0 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.879+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa188004eb0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2861104093 0 0) 0x7fa188004eb0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a4162820 con 0x7fa1a414fdc0 2024-03-29T11:43:46.890 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa19c0019b0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:46.891 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3173803820 0 0) 0x7fa1a4162820 con 0x7fa1a414fdc0 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 >> v1:172.21.15.33:6790/0 conn(0x7fa1a40ad6c0 legacy=0x7fa1a40711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.880+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 >> v1:172.21.15.33:6789/0 conn(0x7fa1a40af6c0 legacy=0x7fa1a4160cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.882+0000 7fa199ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.882+0000 7fa199ffb640 1 monclient: found mon.b 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.882+0000 7fa199ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.882+0000 7fa199ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.882+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a41438d0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.882+0000 7fa199ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa199ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa199ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 10 monclient: _renew_subs 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a4143b20 con 0x7fa1a414fdc0 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:46.892 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 10 monclient: _renew_subs 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a41615a0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa1ab815640 1 librados: init done 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.883+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa19c004e10 con 0x7fa1a414fdc0 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.884+0000 7fa199ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.884+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fa19c0051e0 con 0x7fa1a414fdc0 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.884+0000 7fa199ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.884+0000 7fa199ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.884+0000 7fa199ffb640 10 monclient: dump: 2024-03-29T11:43:46.893 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.884+0000 7fa19b7fe640 4 set_mon_vals no callback set 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.885+0000 7fa1ab815640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.885+0000 7fa1ab815640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.885+0000 7fa1ab815640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:46.894 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.885+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa184005080 con 0x7fa1a414fdc0 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.885+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fa19c005340 con 0x7fa1a414fdc0 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.886+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fa19c085070 con 0x7fa1a414fdc0 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.886+0000 7fa199ffb640 10 client.4570.objecter ms_dispatch 0x7fa1a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.886+0000 7fa199ffb640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.886+0000 7fa199ffb640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.886+0000 7fa199ffb640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.887+0000 7fa1a95b1640 10 monclient: get_auth_request con 0x7fa18806dc30 auth_method 0 2024-03-29T11:43:46.895 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.887+0000 7fa199ffb640 10 client.4570.objecter ms_handle_connect 0x7fa18806dc30 2024-03-29T11:43:46.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.894+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa19c050950 con 0x7fa1a414fdc0 2024-03-29T11:43:46.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.894+0000 7fa199ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:46.899 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:46.894+0000 7fa199ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:47.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.234+0000 7fa1ab815640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:47.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.234+0000 7fa1ab815640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:47.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.234+0000 7fa1ab815640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:47.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.234+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa184005330 con 0x7fa1a414fdc0 2024-03-29T11:43:47.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.236+0000 7fa199ffb640 1 -- 172.21.15.33:0/1789904874 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fa19c052880 con 0x7fa1a414fdc0 2024-03-29T11:43:47.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.236+0000 7fa199ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:47.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.236+0000 7fa199ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:47.239 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:47.240 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 10 librados: watch_flush enter 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 10 librados: watch_flush exit 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 10 client.4570.objecter successfully canceled tick 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 >> v1:172.21.15.153:6816/2737028074 conn(0x7fa18806dc30 legacy=0x7fa1880700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 10 monclient: shutdown 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 >> v1:172.21.15.153:6789/0 conn(0x7fa1a414fdc0 legacy=0x7fa1a41f74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 shutdown_connections 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.240+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 >> 172.21.15.33:0/1789904874 conn(0x7fa1a40b35e0 msgr2=0x7fa1a40a9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:47.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.241+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 shutdown_connections 2024-03-29T11:43:47.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.241+0000 7fa1ab815640 1 -- 172.21.15.33:0/1789904874 wait complete. 2024-03-29T11:43:47.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:47.241+0000 7fa1ab815640 1 librados: shutdown 2024-03-29T11:43:47.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:54.067 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:54.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.217+0000 7f1224332640 10 monclient: get_monmap_and_config 2024-03-29T11:43:54.220 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.217+0000 7f1224332640 10 monclient: build_initial_monmap 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.217+0000 7f1224332640 10 monclient: monmap: 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:43:54.217851+0000 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:43:54.217851+0000 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:54.221 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:54.222 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:43:54.222 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:43:54.222 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.218+0000 7f1224332640 1 Processor -- start 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.218+0000 7f1224332640 1 -- start start 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.218+0000 7f1224332640 10 monclient: init 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 10 monclient(hunting): picked mon.noname-b con 0x7f121c0b1f60 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 10 monclient(hunting): picked mon.noname-c con 0x7f121c071eb0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 10 monclient(hunting): picked mon.noname-a con 0x7f121c143d10 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c06ef80 con 0x7f121c143d10 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c06f260 con 0x7f121c071eb0 2024-03-29T11:43:54.223 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c0b40c0 con 0x7f121c0b1f60 2024-03-29T11:43:54.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:54.224 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.219+0000 7f1224332640 10 monclient(hunting): authenticate will time out at 1390.483219s 2024-03-29T11:43:54.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12228cf640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f121c143d10 0x7f121c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:53605/0 (socket says 172.21.15.33:53605) 2024-03-29T11:43:54.226 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12228cf640 1 -- 172.21.15.33:0/4130089949 learned_addr learned my addr 172.21.15.33:0/4130089949 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212916756 0 0) 0x7f121c06ef80 con 0x7f121c143d10 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12210cc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12210cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12210cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12210cc640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1204003520 con 0x7f121c143d10 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.223+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2386588162 0 0) 0x7f1204003520 con 0x7f121c143d10 2024-03-29T11:43:54.227 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 >> v1:172.21.15.33:6790/0 conn(0x7f121c0b1f60 legacy=0x7f121c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 >> v1:172.21.15.153:6789/0 conn(0x7f121c071eb0 legacy=0x7f121c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 1 monclient: found mon.noname-a 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c155d70 con 0x7f121c143d10 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: _finish_auth 0 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:54.228 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f120c001670 con 0x7f121c143d10 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: dump: 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:54.229 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:54.230 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:54.230 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f1224332640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:43:54.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f1224332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:54.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f120c0018d0 con 0x7f121c143d10 2024-03-29T11:43:54.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:54.232 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f1224332640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f1224332640 4 set_mon_vals no callback set 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 1 -- 172.21.15.33:0/4130089949 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f120c001670 con 0x7f121c143d10 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.224+0000 7f12210cc640 10 monclient: dump: 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:54.233 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.225+0000 7f1224332640 10 monclient: shutdown 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.225+0000 7f1224332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.225+0000 7f1224332640 1 -- 172.21.15.33:0/4130089949 >> v1:172.21.15.33:6789/0 conn(0x7f121c143d10 legacy=0x7f121c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.225+0000 7f1224332640 1 -- 172.21.15.33:0/4130089949 shutdown_connections 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.225+0000 7f1224332640 1 -- 172.21.15.33:0/4130089949 >> 172.21.15.33:0/4130089949 conn(0x7f121c066a00 msgr2=0x7f121c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.226+0000 7f1224332640 1 -- 172.21.15.33:0/4130089949 shutdown_connections 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.226+0000 7f1224332640 1 -- 172.21.15.33:0/4130089949 wait complete. 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.227+0000 7f1224332640 10 monclient: build_initial_monmap 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.227+0000 7f1224332640 10 monclient: monmap: 2024-03-29T11:43:54.234 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.227+0000 7f1224332640 1 librados: starting msgr at 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.227+0000 7f1224332640 1 librados: starting objecter 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.227+0000 7f1224332640 1 Processor -- start 2024-03-29T11:43:54.235 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.228+0000 7f1224332640 1 -- start start 2024-03-29T11:43:54.236 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.228+0000 7f1224332640 1 librados: setting wanted keys 2024-03-29T11:43:54.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.228+0000 7f1224332640 1 librados: calling monclient init 2024-03-29T11:43:54.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.228+0000 7f1224332640 10 monclient: init 2024-03-29T11:43:54.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.229+0000 7f1224332640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:54.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.229+0000 7f1224332640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:54.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.229+0000 7f1224332640 10 monclient(hunting): picked mon.noname-a con 0x7f121c0b1f60 addr v1:172.21.15.33:6789/0 2024-03-29T11:43:54.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.229+0000 7f1224332640 10 monclient(hunting): picked mon.noname-c con 0x7f121c143d10 addr v1:172.21.15.33:6790/0 2024-03-29T11:43:54.237 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f1224332640 10 monclient(hunting): picked mon.noname-b con 0x7f121c1492d0 addr v1:172.21.15.153:6789/0 2024-03-29T11:43:54.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f1224332640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c149b50 con 0x7f121c0b1f60 2024-03-29T11:43:54.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f12228cf640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f121c1492d0 0x7f121c1e60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:51841/0 (socket says 172.21.15.33:51841) 2024-03-29T11:43:54.238 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f12228cf640 1 -- 172.21.15.33:0/3358172171 learned_addr learned my addr 172.21.15.33:0/3358172171 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:43:54.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c14b1a0 con 0x7f121c1492d0 2024-03-29T11:43:54.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c14b480 con 0x7f121c143d10 2024-03-29T11:43:54.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f1224332640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:54.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f1224332640 10 monclient(hunting): authenticate will time out at 1390.496945s 2024-03-29T11:43:54.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 10 client.?.objecter ms_handle_connect 0x7f121c1492d0 2024-03-29T11:43:54.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765373836 0 0) 0x7f121c14b1a0 con 0x7f121c1492d0 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.233+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1208004eb0 con 0x7f121c1492d0 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1579254863 0 0) 0x7f1208004eb0 con 0x7f121c1492d0 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f121c14b1a0 con 0x7f121c1492d0 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f120c001670 con 0x7f121c1492d0 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:54.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 10 monclient(hunting): dump: 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:54.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1404407321 0 0) 0x7f121c14b1a0 con 0x7f121c1492d0 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 >> v1:172.21.15.33:6790/0 conn(0x7f121c143d10 legacy=0x7f121c147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.235+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 >> v1:172.21.15.33:6789/0 conn(0x7f121c0b1f60 legacy=0x7f121c146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 1 monclient: found mon.b 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c147ea0 con 0x7f121c1492d0 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: _finish_auth 0 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 client.?.objecter ms_handle_connect 0x7f121c0b1f60 2024-03-29T11:43:54.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 10 monclient: _renew_subs 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121c149750 con 0x7f121c1492d0 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 10 monclient: _renew_subs 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c14a1c0 con 0x7f121c1492d0 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f1224332640 1 librados: init done 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f120c004a90 con 0x7f121c1492d0 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f120c004ea0 con 0x7f121c1492d0 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:54.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121affd640 10 monclient: dump: 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:43:54.246 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.236+0000 7f121bfff640 4 set_mon_vals no callback set 2024-03-29T11:43:54.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f1224332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:54.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f1224332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:54.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f1224332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:54.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc005080 con 0x7f121c1492d0 2024-03-29T11:43:54.247 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f120c005000 con 0x7f121c1492d0 2024-03-29T11:43:54.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f120c084cf0 con 0x7f121c1492d0 2024-03-29T11:43:54.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f121affd640 10 client.4573.objecter ms_dispatch 0x7f121c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:54.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f121affd640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:54.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f121affd640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:54.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.238+0000 7f121affd640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-29T11:43:54.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.239+0000 7f12220ce640 10 monclient: get_auth_request con 0x7f120806dc10 auth_method 0 2024-03-29T11:43:54.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.240+0000 7f121affd640 10 client.4573.objecter ms_handle_connect 0x7f120806dc10 2024-03-29T11:43:54.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.246+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f120c0505d0 con 0x7f121c1492d0 2024-03-29T11:43:54.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.246+0000 7f121affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:54.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.246+0000 7f121affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:54.574 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.570+0000 7f1224332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:54.574 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.570+0000 7f1224332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:54.574 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.570+0000 7f1224332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:43:54.574 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.570+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11fc005330 con 0x7f121c1492d0 2024-03-29T11:43:54.577 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:43:54.578 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:54.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.574+0000 7f121affd640 1 -- 172.21.15.33:0/3358172171 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f120c052500 con 0x7f121c1492d0 2024-03-29T11:43:54.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.574+0000 7f121affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:54.578 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.574+0000 7f121affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.577+0000 7f1224332640 10 librados: watch_flush enter 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.577+0000 7f1224332640 10 librados: watch_flush exit 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 10 client.4573.objecter successfully canceled tick 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 >> v1:172.21.15.153:6816/2737028074 conn(0x7f120806dc10 legacy=0x7f12080700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 10 monclient: shutdown 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 >> v1:172.21.15.153:6789/0 conn(0x7f121c1492d0 legacy=0x7f121c1e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 shutdown_connections 2024-03-29T11:43:54.581 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 >> 172.21.15.33:0/3358172171 conn(0x7f121c066a00 msgr2=0x7f121c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:54.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 shutdown_connections 2024-03-29T11:43:54.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.578+0000 7f1224332640 1 -- 172.21.15.33:0/3358172171 wait complete. 2024-03-29T11:43:54.583 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:43:54.580+0000 7f1224332640 1 librados: shutdown 2024-03-29T11:43:54.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:01.412 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:01.573 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.570+0000 7f1b37006640 10 monclient: get_monmap_and_config 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.570+0000 7f1b37006640 10 monclient: build_initial_monmap 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.570+0000 7f1b37006640 10 monclient: monmap: 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:01.571117+0000 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:01.571117+0000 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:01.574 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.571+0000 7f1b37006640 1 Processor -- start 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.571+0000 7f1b37006640 1 -- start start 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.571+0000 7f1b37006640 10 monclient: init 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.572+0000 7f1b37006640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.572+0000 7f1b37006640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.572+0000 7f1b37006640 10 monclient(hunting): picked mon.noname-b con 0x7f1b300b1f60 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.572+0000 7f1b37006640 10 monclient(hunting): picked mon.noname-c con 0x7f1b30071eb0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:01.576 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.573+0000 7f1b37006640 10 monclient(hunting): picked mon.noname-a con 0x7f1b30143d10 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:01.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.573+0000 7f1b37006640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3006ef80 con 0x7f1b30143d10 2024-03-29T11:44:01.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.573+0000 7f1b37006640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3006f260 con 0x7f1b30071eb0 2024-03-29T11:44:01.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.573+0000 7f1b37006640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b300b40c0 con 0x7f1b300b1f60 2024-03-29T11:44:01.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.573+0000 7f1b37006640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:01.577 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.573+0000 7f1b37006640 10 monclient(hunting): authenticate will time out at 1397.836922s 2024-03-29T11:44:01.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.578+0000 7f1b355a3640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f1b30143d10 0x7f1b30146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:38687/0 (socket says 172.21.15.33:38687) 2024-03-29T11:44:01.582 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.578+0000 7f1b355a3640 1 -- 172.21.15.33:0/1128933000 learned_addr learned my addr 172.21.15.33:0/1128933000 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:01.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.579+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092139562 0 0) 0x7f1b3006ef80 con 0x7f1b30143d10 2024-03-29T11:44:01.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.579+0000 7f1b2f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:01.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.579+0000 7f1b2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:01.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.579+0000 7f1b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:01.584 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.579+0000 7f1b2f7fe640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.579+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b20003520 con 0x7f1b30143d10 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350815035 0 0) 0x7f1b20003520 con 0x7f1b30143d10 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 >> v1:172.21.15.33:6790/0 conn(0x7f1b300b1f60 legacy=0x7f1b30143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 >> v1:172.21.15.153:6789/0 conn(0x7f1b30071eb0 legacy=0x7f1b30146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30155d70 con 0x7f1b30143d10 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1b28001a10 con 0x7f1b30143d10 2024-03-29T11:44:01.585 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b2f7fe640 10 monclient: dump: 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b37006640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:44:01.586 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.580+0000 7f1b37006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.582+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b28001ca0 con 0x7f1b30143d10 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.582+0000 7f1b2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.582+0000 7f1b2f7fe640 1 -- 172.21.15.33:0/1128933000 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1b28002030 con 0x7f1b30143d10 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.582+0000 7f1b2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.582+0000 7f1b2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.582+0000 7f1b2f7fe640 10 monclient: dump: 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:01.587 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.583+0000 7f1b37006640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.583+0000 7f1b37006640 4 set_mon_vals no callback set 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.584+0000 7f1b37006640 10 monclient: shutdown 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.584+0000 7f1b37006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.584+0000 7f1b37006640 1 -- 172.21.15.33:0/1128933000 >> v1:172.21.15.33:6789/0 conn(0x7f1b30143d10 legacy=0x7f1b30146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.584+0000 7f1b37006640 1 -- 172.21.15.33:0/1128933000 shutdown_connections 2024-03-29T11:44:01.588 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.584+0000 7f1b37006640 1 -- 172.21.15.33:0/1128933000 >> 172.21.15.33:0/1128933000 conn(0x7f1b30066a00 msgr2=0x7f1b30066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:01.590 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.587+0000 7f1b37006640 1 -- 172.21.15.33:0/1128933000 shutdown_connections 2024-03-29T11:44:01.593 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.589+0000 7f1b37006640 1 -- 172.21.15.33:0/1128933000 wait complete. 2024-03-29T11:44:01.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.591+0000 7f1b37006640 10 monclient: build_initial_monmap 2024-03-29T11:44:01.594 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.591+0000 7f1b37006640 10 monclient: monmap: 2024-03-29T11:44:01.594 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:01.594 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:01.594 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:01.594 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:01.594 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:01.595 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.595 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:01.595 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:01.595 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:01.595 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.592+0000 7f1b37006640 1 librados: starting msgr at 2024-03-29T11:44:01.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.592+0000 7f1b37006640 1 librados: starting objecter 2024-03-29T11:44:01.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.594+0000 7f1b37006640 1 Processor -- start 2024-03-29T11:44:01.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.594+0000 7f1b37006640 1 -- start start 2024-03-29T11:44:01.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.594+0000 7f1b37006640 1 librados: setting wanted keys 2024-03-29T11:44:01.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.594+0000 7f1b37006640 1 librados: calling monclient init 2024-03-29T11:44:01.597 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.594+0000 7f1b37006640 10 monclient: init 2024-03-29T11:44:01.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.595+0000 7f1b37006640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:01.598 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.595+0000 7f1b37006640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:01.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 10 monclient(hunting): picked mon.noname-b con 0x7f1b300b1f60 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:01.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 10 monclient(hunting): picked mon.noname-a con 0x7f1b30143d10 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:01.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 10 monclient(hunting): picked mon.noname-c con 0x7f1b3014a270 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:01.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b30149de0 con 0x7f1b30143d10 2024-03-29T11:44:01.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3014b0e0 con 0x7f1b300b1f60 2024-03-29T11:44:01.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3013bc00 con 0x7f1b3014a270 2024-03-29T11:44:01.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:01.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.596+0000 7f1b37006640 10 monclient(hunting): authenticate will time out at 1397.860073s 2024-03-29T11:44:01.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.600+0000 7f1b2ffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f1b30143d10 0x7f1b3013ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:39203/0 (socket says 172.21.15.33:39203) 2024-03-29T11:44:01.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.600+0000 7f1b2ffff640 1 -- 172.21.15.33:0/4058630069 learned_addr learned my addr 172.21.15.33:0/4058630069 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:01.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 client.?.objecter ms_handle_connect 0x7f1b300b1f60 2024-03-29T11:44:01.604 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 client.?.objecter ms_handle_connect 0x7f1b30143d10 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785297658 0 0) 0x7f1b3014b0e0 con 0x7f1b300b1f60 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:44:01.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.601+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b14004eb0 con 0x7f1b300b1f60 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 client.?.objecter ms_handle_connect 0x7f1b3014a270 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838640747 0 0) 0x7f1b30149de0 con 0x7f1b30143d10 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3014b0e0 con 0x7f1b30143d10 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256379727 0 0) 0x7f1b3013bc00 con 0x7f1b3014a270 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:01.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30149de0 con 0x7f1b3014a270 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688765176 0 0) 0x7f1b14004eb0 con 0x7f1b300b1f60 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b3013bc00 con 0x7f1b300b1f60 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1b240018b0 con 0x7f1b300b1f60 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): dump: 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.612 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.604+0000 7f1b2d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1255425829 0 0) 0x7f1b3014b0e0 con 0x7f1b30143d10 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b14004eb0 con 0x7f1b30143d10 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1b1c001970 con 0x7f1b30143d10 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient(hunting): dump: 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.613 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477834779 0 0) 0x7f1b30149de0 con 0x7f1b3014a270 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b3014b0e0 con 0x7f1b3014a270 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1b28001830 con 0x7f1b3014a270 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient(hunting): dump: 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:01.614 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4150698669 0 0) 0x7f1b14004eb0 con 0x7f1b30143d10 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 >> v1:172.21.15.33:6790/0 conn(0x7f1b3014a270 legacy=0x7f1b3013b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 >> v1:172.21.15.153:6789/0 conn(0x7f1b300b1f60 legacy=0x7f1b30147770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 monclient: found mon.a 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:01.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3014b350 con 0x7f1b30143d10 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.605+0000 7f1b37006640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 10 monclient: _renew_subs 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b30149a70 con 0x7f1b30143d10 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b1c003610 con 0x7f1b30143d10 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 10 monclient: _renew_subs 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b30145890 con 0x7f1b30143d10 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b37006640 1 librados: init done 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:01.616 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1b1c0039a0 con 0x7f1b30143d10 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b2d7fa640 10 monclient: dump: 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:01.617 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.606+0000 7f1b2effd640 4 set_mon_vals no callback set 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.608+0000 7f1b37006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.608+0000 7f1b37006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.608+0000 7f1b37006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.608+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b10005080 con 0x7f1b30143d10 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.608+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f1b1c003b80 con 0x7f1b30143d10 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.609+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f1b1c084ac0 con 0x7f1b30143d10 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.609+0000 7f1b2d7fa640 10 client.4551.objecter ms_dispatch 0x7f1b30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.609+0000 7f1b2d7fa640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.609+0000 7f1b2d7fa640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.609+0000 7f1b2d7fa640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.610+0000 7f1b34da2640 10 monclient: get_auth_request con 0x7f1b1406e150 auth_method 0 2024-03-29T11:44:01.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.610+0000 7f1b2d7fa640 10 client.4551.objecter ms_handle_connect 0x7f1b1406e150 2024-03-29T11:44:01.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.621+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b1c0503a0 con 0x7f1b30143d10 2024-03-29T11:44:01.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.621+0000 7f1b2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:01.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.621+0000 7f1b2d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:01.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.952+0000 7f1b37006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:01.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.952+0000 7f1b37006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:01.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.952+0000 7f1b37006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:01.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.952+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b10005330 con 0x7f1b30143d10 2024-03-29T11:44:01.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.953+0000 7f1b2d7fa640 1 -- 172.21.15.33:0/4058630069 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f1b1c0522d0 con 0x7f1b30143d10 2024-03-29T11:44:01.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.953+0000 7f1b2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:01.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.953+0000 7f1b2d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:01.957 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:01.957 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:01.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.957+0000 7f1b37006640 10 librados: watch_flush enter 2024-03-29T11:44:01.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.959+0000 7f1b37006640 10 librados: watch_flush exit 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.959+0000 7f1b37006640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.959+0000 7f1b37006640 10 client.4551.objecter successfully canceled tick 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.960+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 >> v1:172.21.15.153:6816/2737028074 conn(0x7f1b1406e150 legacy=0x7f1b14070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.960+0000 7f1b37006640 10 monclient: shutdown 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.960+0000 7f1b37006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.960+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 >> v1:172.21.15.33:6789/0 conn(0x7f1b30143d10 legacy=0x7f1b3013ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.960+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 shutdown_connections 2024-03-29T11:44:01.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.960+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 >> 172.21.15.33:0/4058630069 conn(0x7f1b30066a00 msgr2=0x7f1b300b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:01.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.961+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 shutdown_connections 2024-03-29T11:44:01.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.961+0000 7f1b37006640 1 -- 172.21.15.33:0/4058630069 wait complete. 2024-03-29T11:44:01.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:01.961+0000 7f1b37006640 1 librados: shutdown 2024-03-29T11:44:01.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:08.788 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:08.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.930+0000 7f95c93e2640 10 monclient: get_monmap_and_config 2024-03-29T11:44:08.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.930+0000 7f95c93e2640 10 monclient: build_initial_monmap 2024-03-29T11:44:08.935 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.930+0000 7f95c93e2640 10 monclient: monmap: 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:08.932051+0000 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:08.932051+0000 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:08.936 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.931+0000 7f95c93e2640 1 Processor -- start 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.931+0000 7f95c93e2640 1 -- start start 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.931+0000 7f95c93e2640 10 monclient: init 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 10 monclient(hunting): picked mon.noname-b con 0x7f95c40a9ac0 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 10 monclient(hunting): picked mon.noname-a con 0x7f95c413ee90 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 10 monclient(hunting): picked mon.noname-c con 0x7f95c40af900 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c40afd10 con 0x7f95c413ee90 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c40af6f0 con 0x7f95c40af900 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c40aac80 con 0x7f95c40a9ac0 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:08.938 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.932+0000 7f95c93e2640 10 monclient(hunting): authenticate will time out at 1405.197367s 2024-03-29T11:44:08.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c2ffd640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f95c40a9ac0 0x7f95c40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:59495/0 (socket says 172.21.15.33:59495) 2024-03-29T11:44:08.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c2ffd640 1 -- 172.21.15.33:0/2647127546 learned_addr learned my addr 172.21.15.33:0/2647127546 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:08.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c27fc640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f95c413ee90 0x7f95c413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:53108/0 (socket says 172.21.15.33:53108) 2024-03-29T11:44:08.946 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550111360 0 0) 0x7f95c40afd10 con 0x7f95c413ee90 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ac003520 con 0x7f95c413ee90 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090680916 0 0) 0x7f95c40aac80 con 0x7f95c40a9ac0 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.936+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ac003a10 con 0x7f95c40a9ac0 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.937+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1376313342 0 0) 0x7f95ac003520 con 0x7f95c413ee90 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.937+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 >> v1:172.21.15.33:6790/0 conn(0x7f95c40a9ac0 legacy=0x7f95c40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.947 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.937+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 >> v1:172.21.15.153:6789/0 conn(0x7f95c40af900 legacy=0x7f95c4149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c40aaf60 con 0x7f95c413ee90 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95b4001670 con 0x7f95c413ee90 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c1ffb640 10 monclient: dump: 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.948 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c93e2640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.938+0000 7f95c93e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95b4001850 con 0x7f95c413ee90 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:08.949 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c1ffb640 1 -- 172.21.15.33:0/2647127546 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95b4001c60 con 0x7f95c413ee90 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c1ffb640 10 monclient: dump: 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c93e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:08.950 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c93e2640 4 set_mon_vals no callback set 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.939+0000 7f95c93e2640 10 monclient: shutdown 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.940+0000 7f95c93e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.940+0000 7f95c93e2640 1 -- 172.21.15.33:0/2647127546 >> v1:172.21.15.33:6789/0 conn(0x7f95c413ee90 legacy=0x7f95c413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.940+0000 7f95c93e2640 1 -- 172.21.15.33:0/2647127546 shutdown_connections 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.940+0000 7f95c93e2640 1 -- 172.21.15.33:0/2647127546 >> 172.21.15.33:0/2647127546 conn(0x7f95c40b35e0 msgr2=0x7f95c40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.941+0000 7f95c93e2640 1 -- 172.21.15.33:0/2647127546 shutdown_connections 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.941+0000 7f95c93e2640 1 -- 172.21.15.33:0/2647127546 wait complete. 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.942+0000 7f95c93e2640 10 monclient: build_initial_monmap 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.942+0000 7f95c93e2640 10 monclient: monmap: 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:08.951 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:08.952 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:08.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.942+0000 7f95c93e2640 1 librados: starting msgr at 2024-03-29T11:44:08.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.942+0000 7f95c93e2640 1 librados: starting objecter 2024-03-29T11:44:08.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.942+0000 7f95c93e2640 1 Processor -- start 2024-03-29T11:44:08.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.944+0000 7f95c93e2640 1 -- start start 2024-03-29T11:44:08.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.944+0000 7f95c93e2640 1 librados: setting wanted keys 2024-03-29T11:44:08.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.944+0000 7f95c93e2640 1 librados: calling monclient init 2024-03-29T11:44:08.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.944+0000 7f95c93e2640 10 monclient: init 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 10 monclient(hunting): picked mon.noname-a con 0x7f95c40a9ac0 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 10 monclient(hunting): picked mon.noname-b con 0x7f95c40af900 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 10 monclient(hunting): picked mon.noname-c con 0x7f95c413ee90 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4140470 con 0x7f95c40a9ac0 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4140900 con 0x7f95c40af900 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4141820 con 0x7f95c413ee90 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.945+0000 7f95c93e2640 10 monclient(hunting): authenticate will time out at 1405.210089s 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.949+0000 7f95c2ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f95c40a9ac0 0x7f95c406e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:56968/0 (socket says 172.21.15.33:56968) 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.949+0000 7f95c2ffd640 1 -- 172.21.15.33:0/4289470573 learned_addr learned my addr 172.21.15.33:0/4289470573 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.949+0000 7f95ab7fe640 10 client.?.objecter ms_handle_connect 0x7f95c40a9ac0 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.949+0000 7f95ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 10 client.?.objecter ms_handle_connect 0x7f95c40af900 2024-03-29T11:44:08.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:08.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400269373 0 0) 0x7f95c4140470 con 0x7f95c40a9ac0 2024-03-29T11:44:08.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:08.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:08.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:08.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:44:08.957 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.950+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b0004eb0 con 0x7f95c40a9ac0 2024-03-29T11:44:08.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.951+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124696434 0 0) 0x7f95c4140900 con 0x7f95c40af900 2024-03-29T11:44:08.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.951+0000 7f95ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.951+0000 7f95ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c4140470 con 0x7f95c40af900 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3994132752 0 0) 0x7f95c4140470 con 0x7f95c40af900 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95c4140900 con 0x7f95c40af900 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95b4001b60 con 0x7f95c40af900 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:08.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.952+0000 7f95ab7fe640 10 monclient(hunting): dump: 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.953+0000 7f95ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.953+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1040833576 0 0) 0x7f95c4140900 con 0x7f95c40af900 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.953+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 >> v1:172.21.15.33:6790/0 conn(0x7f95c413ee90 legacy=0x7f95c41e6640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 >> v1:172.21.15.33:6789/0 conn(0x7f95c40a9ac0 legacy=0x7f95c406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:08.964 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 1 monclient: found mon.b 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4141b00 con 0x7f95c40af900 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95c93e2640 5 monclient: authenticate success, global_id 4582 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95c93e2640 10 monclient: _renew_subs 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95c93e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.954+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c4141de0 con 0x7f95c40af900 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.955+0000 7f95c93e2640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.955+0000 7f95c93e2640 10 monclient: _renew_subs 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.955+0000 7f95c93e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.955+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c406eeb0 con 0x7f95c40af900 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.955+0000 7f95c93e2640 1 librados: init done 2024-03-29T11:44:08.965 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.956+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95b4001cc0 con 0x7f95c40af900 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.956+0000 7f95ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.956+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f95b4004c30 con 0x7f95c40af900 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.956+0000 7f95ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.956+0000 7f95ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.956+0000 7f95ab7fe640 10 monclient: dump: 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:08.966 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:08.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.957+0000 7f95c17fa640 4 set_mon_vals no callback set 2024-03-29T11:44:08.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.958+0000 7f95c93e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:08.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.958+0000 7f95c93e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:08.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.958+0000 7f95c93e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:08.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.958+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a4005080 con 0x7f95c40af900 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.958+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f95b401ac00 con 0x7f95c40af900 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.960+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f95b4084ab0 con 0x7f95c40af900 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.960+0000 7f95ab7fe640 10 client.4582.objecter ms_dispatch 0x7f95c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.960+0000 7f95c2ffd640 10 monclient: get_auth_request con 0x7f95b006daf0 auth_method 0 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.960+0000 7f95ab7fe640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.960+0000 7f95ab7fe640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.960+0000 7f95ab7fe640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-29T11:44:08.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.960+0000 7f95ab7fe640 10 client.4582.objecter ms_handle_connect 0x7f95b006daf0 2024-03-29T11:44:08.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.966+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95b4050390 con 0x7f95c40af900 2024-03-29T11:44:08.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.966+0000 7f95ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:08.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:08.966+0000 7f95ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:09.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.299+0000 7f95c93e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:09.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.299+0000 7f95c93e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:09.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.299+0000 7f95c93e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:09.304 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.299+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a4005330 con 0x7f95c40af900 2024-03-29T11:44:09.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.302+0000 7f95ab7fe640 1 -- 172.21.15.33:0/4289470573 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f95b40522c0 con 0x7f95c40af900 2024-03-29T11:44:09.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.302+0000 7f95ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:09.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.302+0000 7f95ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:09.307 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:09.308 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:09.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.306+0000 7f95c93e2640 10 librados: watch_flush enter 2024-03-29T11:44:09.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.306+0000 7f95c93e2640 10 librados: watch_flush exit 2024-03-29T11:44:09.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.306+0000 7f95c93e2640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-29T11:44:09.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.306+0000 7f95c93e2640 10 client.4582.objecter successfully canceled tick 2024-03-29T11:44:09.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.308+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 >> v1:172.21.15.153:6816/2737028074 conn(0x7f95b006daf0 legacy=0x7f95b006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:09.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.308+0000 7f95c93e2640 10 monclient: shutdown 2024-03-29T11:44:09.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.308+0000 7f95c93e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:09.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.308+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 >> v1:172.21.15.153:6789/0 conn(0x7f95c40af900 legacy=0x7f95c413c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:09.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.309+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 shutdown_connections 2024-03-29T11:44:09.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.309+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 >> 172.21.15.33:0/4289470573 conn(0x7f95c40b35e0 msgr2=0x7f95c40ad1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:09.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.309+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 shutdown_connections 2024-03-29T11:44:09.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.309+0000 7f95c93e2640 1 -- 172.21.15.33:0/4289470573 wait complete. 2024-03-29T11:44:09.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:09.312+0000 7f95c93e2640 1 librados: shutdown 2024-03-29T11:44:09.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:16.141 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:16.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.281+0000 7f2ebe08b640 10 monclient: get_monmap_and_config 2024-03-29T11:44:16.311 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.281+0000 7f2ebe08b640 10 monclient: build_initial_monmap 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.282+0000 7f2ebe08b640 10 monclient: monmap: 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:16.283583+0000 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:16.283583+0000 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:16.312 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.282+0000 7f2ebe08b640 1 Processor -- start 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.282+0000 7f2ebe08b640 1 -- start start 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.282+0000 7f2ebe08b640 10 monclient: init 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.283+0000 7f2ebe08b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.283+0000 7f2ebe08b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.283+0000 7f2ebe08b640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb806c1e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.283+0000 7f2ebe08b640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb806c5c0 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2ebe08b640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb80ae590 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:16.313 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2ebe08b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb813fc80 con 0x7f2eb806c5c0 2024-03-29T11:44:16.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2ebe08b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb806d5c0 con 0x7f2eb80ae590 2024-03-29T11:44:16.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2ebe08b640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8146430 con 0x7f2eb806c1e0 2024-03-29T11:44:16.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2ebe08b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:16.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2eb6ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f2eb806c5c0 0x7f2eb813ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49470/0 (socket says 172.21.15.33:49470) 2024-03-29T11:44:16.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2eb6ffd640 1 -- 172.21.15.33:0/2378194570 learned_addr learned my addr 172.21.15.33:0/2378194570 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:16.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.284+0000 7f2ebe08b640 10 monclient(hunting): authenticate will time out at 1412.549219s 2024-03-29T11:44:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120108544 0 0) 0x7f2eb813fc80 con 0x7f2eb806c5c0 2024-03-29T11:44:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:16.328 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea0003520 con 0x7f2eb806c5c0 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816197457 0 0) 0x7f2eb806d5c0 con 0x7f2eb80ae590 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb813fc80 con 0x7f2eb80ae590 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813009186 0 0) 0x7f2eb8146430 con 0x7f2eb806c1e0 2024-03-29T11:44:16.329 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.287+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb806d5c0 con 0x7f2eb806c1e0 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2413098296 0 0) 0x7f2eb813fc80 con 0x7f2eb80ae590 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 >> v1:172.21.15.33:6790/0 conn(0x7f2eb806c1e0 legacy=0x7f2eb8145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 >> v1:172.21.15.33:6789/0 conn(0x7f2eb806c5c0 legacy=0x7f2eb813ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.330 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 monclient: found mon.noname-c 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80ade90 con 0x7f2eb80ae590 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:16.331 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2eb00018a0 con 0x7f2eb80ae590 2024-03-29T11:44:16.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:16.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:16.332 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: dump: 2024-03-29T11:44:16.332 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:16.332 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:16.332 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2ebe08b640 5 monclient: authenticate success, global_id 4585 2024-03-29T11:44:16.333 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2ebe08b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb0001ac0 con 0x7f2eb80ae590 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 1 -- 172.21.15.33:0/2378194570 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2eb0001e50 con 0x7f2eb80ae590 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.288+0000 7f2eb67fc640 10 monclient: dump: 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:16.334 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.335 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.335 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:16.335 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:16.335 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:16.335 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.289+0000 7f2ebe08b640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.289+0000 7f2ebe08b640 4 set_mon_vals no callback set 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.290+0000 7f2ebe08b640 10 monclient: shutdown 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.290+0000 7f2ebe08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.290+0000 7f2ebe08b640 1 -- 172.21.15.33:0/2378194570 >> v1:172.21.15.153:6789/0 conn(0x7f2eb80ae590 legacy=0x7f2eb813f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.336 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.290+0000 7f2ebe08b640 1 -- 172.21.15.33:0/2378194570 shutdown_connections 2024-03-29T11:44:16.337 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.290+0000 7f2ebe08b640 1 -- 172.21.15.33:0/2378194570 >> 172.21.15.33:0/2378194570 conn(0x7f2eb8067500 msgr2=0x7f2eb80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:16.337 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.290+0000 7f2ebe08b640 1 -- 172.21.15.33:0/2378194570 shutdown_connections 2024-03-29T11:44:16.337 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.290+0000 7f2ebe08b640 1 -- 172.21.15.33:0/2378194570 wait complete. 2024-03-29T11:44:16.337 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.291+0000 7f2ebe08b640 10 monclient: build_initial_monmap 2024-03-29T11:44:16.337 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.291+0000 7f2ebe08b640 10 monclient: monmap: 2024-03-29T11:44:16.337 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:16.337 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:16.338 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.291+0000 7f2ebe08b640 1 librados: starting msgr at 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.291+0000 7f2ebe08b640 1 librados: starting objecter 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.292+0000 7f2ebe08b640 1 Processor -- start 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.292+0000 7f2ebe08b640 1 -- start start 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.292+0000 7f2ebe08b640 1 librados: setting wanted keys 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.292+0000 7f2ebe08b640 1 librados: calling monclient init 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.292+0000 7f2ebe08b640 10 monclient: init 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.293+0000 7f2ebe08b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:16.339 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.293+0000 7f2ebe08b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.293+0000 7f2ebe08b640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb806c1e0 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.293+0000 7f2ebe08b640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb806c5c0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2eb6ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2eb806c5c0 0x7f2eb8142130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:43886/0 (socket says 172.21.15.33:43886) 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2eb77fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f2eb806c1e0 0x7f2eb8136480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:60916/0 (socket says 172.21.15.33:60916) 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2ebe08b640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb80ae590 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2eb77fe640 1 -- 172.21.15.33:0/3712685676 learned_addr learned my addr 172.21.15.33:0/3712685676 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8142840 con 0x7f2eb806c1e0 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8141c50 con 0x7f2eb806c5c0 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 client.?.objecter ms_handle_connect 0x7f2eb806c1e0 2024-03-29T11:44:16.340 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 client.?.objecter ms_handle_connect 0x7f2eb806c5c0 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081521807 0 0) 0x7f2eb8142840 con 0x7f2eb806c1e0 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8135d80 con 0x7f2eb80ae590 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2ebe08b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2ebe08b640 10 monclient(hunting): authenticate will time out at 1412.559433s 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:44:16.341 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.294+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4004eb0 con 0x7f2eb806c1e0 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172621389 0 0) 0x7f2eb8141c50 con 0x7f2eb806c5c0 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): my global_id is 4588 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb8142840 con 0x7f2eb806c5c0 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127178325 0 0) 0x7f2ea4004eb0 con 0x7f2eb806c1e0 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eb8141c50 con 0x7f2eb806c1e0 2024-03-29T11:44:16.342 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2ea8001a40 con 0x7f2eb806c1e0 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): dump: 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.343 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1612970013 0 0) 0x7f2eb8141c50 con 0x7f2eb806c1e0 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.295+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 >> v1:172.21.15.33:6790/0 conn(0x7f2eb80ae590 legacy=0x7f2eb81e1f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 >> v1:172.21.15.153:6789/0 conn(0x7f2eb806c5c0 legacy=0x7f2eb8142130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:16.344 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 1 monclient: found mon.a 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8136060 con 0x7f2eb806c1e0 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2e9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 10 monclient: _renew_subs 2024-03-29T11:44:16.345 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb8140f10 con 0x7f2eb806c1e0 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 10 monclient: _renew_subs 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb81413f0 con 0x7f2eb806c1e0 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.296+0000 7f2ebe08b640 1 librados: init done 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ea8001ba0 con 0x7f2eb806c1e0 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2e9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f2ea8003a60 con 0x7f2eb806c1e0 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2e9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:16.346 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2e9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2e9ffff640 10 monclient: dump: 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:16.347 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2eb5ffb640 4 set_mon_vals no callback set 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.297+0000 7f2ebe08b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.298+0000 7f2ebe08b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.298+0000 7f2ebe08b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.298+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e98005080 con 0x7f2eb806c1e0 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.299+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f2ea8003bc0 con 0x7f2eb806c1e0 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.299+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f2ea8084b70 con 0x7f2eb806c1e0 2024-03-29T11:44:16.348 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.299+0000 7f2e9ffff640 10 client.4563.objecter ms_dispatch 0x7f2eb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.299+0000 7f2e9ffff640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.299+0000 7f2e9ffff640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.300+0000 7f2e9ffff640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.300+0000 7f2eb6ffd640 10 monclient: get_auth_request con 0x7f2ea406dad0 auth_method 0 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.300+0000 7f2e9ffff640 10 client.4563.objecter ms_handle_connect 0x7f2ea406dad0 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.307+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ea8050450 con 0x7f2eb806c1e0 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.308+0000 7f2e9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:16.349 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.308+0000 7f2e9ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:16.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.633+0000 7f2ebe08b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:16.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.633+0000 7f2ebe08b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:16.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.633+0000 7f2ebe08b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:16.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.633+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e98005330 con 0x7f2eb806c1e0 2024-03-29T11:44:16.639 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.634+0000 7f2e9ffff640 1 -- 172.21.15.33:0/3712685676 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f2ea8052380 con 0x7f2eb806c1e0 2024-03-29T11:44:16.639 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.634+0000 7f2e9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:16.639 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.634+0000 7f2e9ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:16.639 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:16.639 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:16.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 10 librados: watch_flush enter 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 10 librados: watch_flush exit 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 >> v1:172.21.15.153:6816/2737028074 conn(0x7f2ea406dad0 legacy=0x7f2ea406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 10 monclient: shutdown 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.637+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 >> v1:172.21.15.33:6789/0 conn(0x7f2eb806c1e0 legacy=0x7f2eb8136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:16.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.638+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 shutdown_connections 2024-03-29T11:44:16.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.638+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 >> 172.21.15.33:0/3712685676 conn(0x7f2eb8067500 msgr2=0x7f2eb813e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:16.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.639+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 shutdown_connections 2024-03-29T11:44:16.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.639+0000 7f2ebe08b640 1 -- 172.21.15.33:0/3712685676 wait complete. 2024-03-29T11:44:16.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:16.639+0000 7f2ebe08b640 1 librados: shutdown 2024-03-29T11:44:16.663 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:23.469 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:23.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.613+0000 7f77b65bc640 10 monclient: get_monmap_and_config 2024-03-29T11:44:23.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.613+0000 7f77b65bc640 10 monclient: build_initial_monmap 2024-03-29T11:44:23.618 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.613+0000 7f77b65bc640 10 monclient: monmap: 2024-03-29T11:44:23.618 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:23.618 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:23.618 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:23.615240+0000 2024-03-29T11:44:23.619 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:23.615240+0000 2024-03-29T11:44:23.619 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:23.619 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.619 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:23.619 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:23.619 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:23.619 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.614+0000 7f77b65bc640 1 Processor -- start 2024-03-29T11:44:23.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.614+0000 7f77b65bc640 1 -- start start 2024-03-29T11:44:23.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.614+0000 7f77b65bc640 10 monclient: init 2024-03-29T11:44:23.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 10 monclient(hunting): picked mon.noname-b con 0x7f77b01457f0 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 10 monclient(hunting): picked mon.noname-c con 0x7f77b013f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 10 monclient(hunting): picked mon.noname-a con 0x7f77b00a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b00aaef0 con 0x7f77b00a9d70 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b0145bd0 con 0x7f77b013f650 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b0145eb0 con 0x7f77b01457f0 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.615+0000 7f77b65bc640 10 monclient(hunting): authenticate will time out at 1419.880510s 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77b4b59640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 0x7f77b014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49604/0 (socket says 172.21.15.33:49604) 2024-03-29T11:44:23.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77b4b59640 1 -- 172.21.15.33:0/2387342491 learned_addr learned my addr 172.21.15.33:0/2387342491 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77af7fe640 1 --1- 172.21.15.33:0/2387342491 >> v1:172.21.15.153:6789/0 conn(0x7f77b013f650 0x7f77b00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:60550/0 (socket says 172.21.15.33:60550) 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408098422 0 0) 0x7f77b00aaef0 con 0x7f77b00a9d70 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7798003520 con 0x7f77b00a9d70 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164075289 0 0) 0x7f77b0145bd0 con 0x7f77b013f650 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:44:23.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b00aaef0 con 0x7f77b013f650 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846236094 0 0) 0x7f7798003520 con 0x7f77b00a9d70 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 >> v1:172.21.15.33:6790/0 conn(0x7f77b01457f0 legacy=0x7f77b013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 >> v1:172.21.15.153:6789/0 conn(0x7f77b013f650 legacy=0x7f77b00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 monclient: found mon.noname-a 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b0146340 con 0x7f77b00a9d70 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f77a8001a10 con 0x7f77b00a9d70 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.616+0000 7f77aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 10 monclient: dump: 2024-03-29T11:44:23.627 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77b65bc640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77b65bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77a8001b70 con 0x7f77b00a9d70 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 1 -- 172.21.15.33:0/2387342491 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f77a8001fc0 con 0x7f77b00a9d70 2024-03-29T11:44:23.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77aeffd640 10 monclient: dump: 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77b65bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.617+0000 7f77b65bc640 4 set_mon_vals no callback set 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.618+0000 7f77b65bc640 10 monclient: shutdown 2024-03-29T11:44:23.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.618+0000 7f77b65bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.618+0000 7f77b65bc640 1 -- 172.21.15.33:0/2387342491 >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 legacy=0x7f77b014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.618+0000 7f77b65bc640 1 -- 172.21.15.33:0/2387342491 shutdown_connections 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.618+0000 7f77b65bc640 1 -- 172.21.15.33:0/2387342491 >> 172.21.15.33:0/2387342491 conn(0x7f77b0066a00 msgr2=0x7f77b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.618+0000 7f77b65bc640 1 -- 172.21.15.33:0/2387342491 shutdown_connections 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.618+0000 7f77b65bc640 1 -- 172.21.15.33:0/2387342491 wait complete. 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.619+0000 7f77b65bc640 10 monclient: build_initial_monmap 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.619+0000 7f77b65bc640 10 monclient: monmap: 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:23.630 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:23.631 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:23.631 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.620+0000 7f77b65bc640 1 librados: starting msgr at 2024-03-29T11:44:23.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.620+0000 7f77b65bc640 1 librados: starting objecter 2024-03-29T11:44:23.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.620+0000 7f77b65bc640 1 Processor -- start 2024-03-29T11:44:23.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.620+0000 7f77b65bc640 1 -- start start 2024-03-29T11:44:23.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.620+0000 7f77b65bc640 1 librados: setting wanted keys 2024-03-29T11:44:23.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.620+0000 7f77b65bc640 1 librados: calling monclient init 2024-03-29T11:44:23.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.620+0000 7f77b65bc640 10 monclient: init 2024-03-29T11:44:23.640 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 10 monclient(hunting): picked mon.noname-a con 0x7f77b00a9d70 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 10 monclient(hunting): picked mon.noname-c con 0x7f77b013f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 10 monclient(hunting): picked mon.noname-b con 0x7f77b01457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b01446a0 con 0x7f77b00a9d70 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b01448f0 con 0x7f77b01457f0 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77affff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 0x7f77b0143700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:49894/0 (socket says 172.21.15.33:49894) 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77affff640 1 -- 172.21.15.33:0/1124151216 learned_addr learned my addr 172.21.15.33:0/1124151216 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b0144bd0 con 0x7f77b013f650 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77b65bc640 10 monclient(hunting): authenticate will time out at 1419.886314s 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77acff9640 10 client.?.objecter ms_handle_connect 0x7f77b01457f0 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77acff9640 10 client.?.objecter ms_handle_connect 0x7f77b013f650 2024-03-29T11:44:23.641 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77acff9640 10 client.?.objecter ms_handle_connect 0x7f77b00a9d70 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.621+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211954868 0 0) 0x7f77b01448f0 con 0x7f77b01457f0 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c004eb0 con 0x7f77b01457f0 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690074129 0 0) 0x7f77b0144bd0 con 0x7f77b013f650 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b01448f0 con 0x7f77b013f650 2024-03-29T11:44:23.642 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211613255 0 0) 0x7f77b01446a0 con 0x7f77b00a9d70 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b0144bd0 con 0x7f77b00a9d70 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3593372252 0 0) 0x7f77b0144bd0 con 0x7f77b00a9d70 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b01446a0 con 0x7f77b00a9d70 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f77a40017d0 con 0x7f77b00a9d70 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): dump: 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.643 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3943730434 0 0) 0x7f77b01448f0 con 0x7f77b013f650 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b0144bd0 con 0x7f77b013f650 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f77a0001930 con 0x7f77b013f650 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.622+0000 7f77acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 10 monclient(hunting): dump: 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.644 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 617822109 0 0) 0x7f77b01446a0 con 0x7f77b00a9d70 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6790/0 conn(0x7f77b013f650 legacy=0x7f77b013c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.153:6789/0 conn(0x7f77b01457f0 legacy=0x7f77b013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 1 monclient: found mon.a 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b013e7c0 con 0x7f77b00a9d70 2024-03-29T11:44:23.645 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77b65bc640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77b65bc640 10 monclient: _renew_subs 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77b65bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b0143e10 con 0x7f77b00a9d70 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.623+0000 7f77b65bc640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77b65bc640 10 monclient: _renew_subs 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77b65bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b013d980 con 0x7f77b00a9d70 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77b65bc640 1 librados: init done 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77a4001930 con 0x7f77b00a9d70 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f77a40037b0 con 0x7f77b00a9d70 2024-03-29T11:44:23.646 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.624+0000 7f77acff9640 10 monclient: dump: 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.625+0000 7f77b65bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.625+0000 7f77b65bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:23.647 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.626+0000 7f77b65bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.626+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7790005080 con 0x7f77b00a9d70 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.626+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f77a4003910 con 0x7f77b00a9d70 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77affff640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 legacy=0x7f77b0143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77affff640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 legacy=0x7f77b0143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77affff640 1 --1- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 0x7f77b0143700 :-1 s=OPENED pgs=950129 cs=1 l=1).handle_message read tag failed 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77affff640 1 --1- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 0x7f77b0143700 :-1 s=OPENED pgs=950129 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.33:6789/0 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6789/0 conn(0x7f77b00a9d70 legacy=0x7f77b0143700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 10 monclient(hunting): picked mon.a con 0x7f779c071a40 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 10 monclient(hunting): picked mon.b con 0x7f77b01457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 10 monclient(hunting): picked mon.c con 0x7f77b013f650 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f77b01446a0 con 0x7f779c071a40 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f779c004eb0 con 0x7f77b01457f0 2024-03-29T11:44:23.648 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f77b01448f0 con 0x7f77b013f650 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.627+0000 7f77acff9640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77affff640 10 monclient(hunting): get_auth_request con 0x7f779c06dd70 auth_method 0 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 client.4569.objecter ms_handle_connect 0x7f779c071a40 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 client.4569.objecter resend_mon_ops 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 client.4569.objecter ms_handle_connect 0x7f779c06dd70 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 client.4569.objecter ms_handle_connect 0x7f77b013f650 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 client.4569.objecter resend_mon_ops 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 client.4569.objecter ms_handle_connect 0x7f77b01457f0 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 client.4569.objecter resend_mon_ops 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279502569 0 0) 0x7f77b01446a0 con 0x7f779c071a40 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f779c07ef70 con 0x7f779c071a40 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897032459 0 0) 0x7f779c004eb0 con 0x7f77b01457f0 2024-03-29T11:44:23.649 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.628+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.153:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f77b01446a0 con 0x7f77b01457f0 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952578073 0 0) 0x7f77b01448f0 con 0x7f77b013f650 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.33:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f779c004eb0 con 0x7f77b013f650 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2539737773 0 0) 0x7f77b01446a0 con 0x7f77b01457f0 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6790/0 conn(0x7f77b013f650 legacy=0x7f779c07b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.33:6789/0 conn(0x7f779c071a40 legacy=0x7f779c073e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 monclient: found mon.b 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:23.650 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f779c07ffa0 con 0x7f77b01457f0 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b01448f0 con 0x7f77b01457f0 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f77a8004cc0 con 0x7f77b01457f0 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: dump: 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:23.651 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77a80020d0 con 0x7f77b01457f0 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.629+0000 7f77adffb640 4 set_mon_vals no callback set 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.631+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f77a801ad10 con 0x7f77b01457f0 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.632+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f77a8084bd0 con 0x7f77b01457f0 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.632+0000 7f77acff9640 10 client.4569.objecter ms_dispatch 0x7f77b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.632+0000 7f77acff9640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.632+0000 7f77acff9640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:23.652 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.632+0000 7f77acff9640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T11:44:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.638+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77a8001ce0 con 0x7f77b01457f0 2024-03-29T11:44:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.638+0000 7f77acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:23.653 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:23.638+0000 7f77acff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:24.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.001+0000 7f77b65bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:24.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.001+0000 7f77b65bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:24.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.001+0000 7f77b65bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:24.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.001+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7790005330 con 0x7f77b01457f0 2024-03-29T11:44:24.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.003+0000 7f77acff9640 1 -- 172.21.15.33:0/1124151216 <== mon.1 v1:172.21.15.153:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f77a8053c00 con 0x7f77b01457f0 2024-03-29T11:44:24.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.003+0000 7f77acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:24.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.003+0000 7f77acff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:24.008 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:24.008 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:24.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 10 librados: watch_flush enter 2024-03-29T11:44:24.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 10 librados: watch_flush exit 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 10 client.4569.objecter successfully canceled tick 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.153:6816/2737028074 conn(0x7f779c06dd70 legacy=0x7f779c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 10 monclient: shutdown 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.006+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 >> v1:172.21.15.153:6789/0 conn(0x7f77b01457f0 legacy=0x7f779c077840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.007+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 shutdown_connections 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.007+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 >> 172.21.15.33:0/1124151216 conn(0x7f77b0066a00 msgr2=0x7f77b013a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.007+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 shutdown_connections 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.007+0000 7f77b65bc640 1 -- 172.21.15.33:0/1124151216 wait complete. 2024-03-29T11:44:24.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:24.007+0000 7f77b65bc640 1 librados: shutdown 2024-03-29T11:44:24.026 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:30.834 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:30.983 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.978+0000 7fed9bd4d640 10 monclient: get_monmap_and_config 2024-03-29T11:44:30.983 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.978+0000 7fed9bd4d640 10 monclient: build_initial_monmap 2024-03-29T11:44:30.983 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.978+0000 7fed9bd4d640 10 monclient: monmap: 2024-03-29T11:44:30.983 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:30.983 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:30.980465+0000 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:30.980465+0000 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:30.984 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:30.986 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.982+0000 7fed9bd4d640 1 Processor -- start 2024-03-29T11:44:30.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.982+0000 7fed9bd4d640 1 -- start start 2024-03-29T11:44:30.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.982+0000 7fed9bd4d640 10 monclient: init 2024-03-29T11:44:30.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:30.988 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 10 monclient(hunting): picked mon.noname-b con 0x7fed940b1f60 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 10 monclient(hunting): picked mon.noname-a con 0x7fed94071eb0 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 10 monclient(hunting): picked mon.noname-c con 0x7fed94143d10 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9406ef80 con 0x7fed94071eb0 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9406f260 con 0x7fed94143d10 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940b40c0 con 0x7fed940b1f60 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.983+0000 7fed9bd4d640 10 monclient(hunting): authenticate will time out at 1427.248624s 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.984+0000 7fed99ae9640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fed940b1f60 0x7fed94143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:48826/0 (socket says 172.21.15.33:48826) 2024-03-29T11:44:30.989 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.984+0000 7fed99ae9640 1 -- 172.21.15.33:0/757750300 learned_addr learned my addr 172.21.15.33:0/757750300 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:30.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.986+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297723133 0 0) 0x7fed940b40c0 con 0x7fed940b1f60 2024-03-29T11:44:30.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.986+0000 7fed98ae7640 10 monclient(hunting): _negotiate 2024-03-29T11:44:30.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.986+0000 7fed98ae7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:30.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.986+0000 7fed98ae7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:30.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.986+0000 7fed98ae7640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:44:30.990 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.986+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed80003520 con 0x7fed940b1f60 2024-03-29T11:44:30.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.987+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1767310803 0 0) 0x7fed80003520 con 0x7fed940b1f60 2024-03-29T11:44:30.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.987+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 >> v1:172.21.15.153:6789/0 conn(0x7fed94143d10 legacy=0x7fed94146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:30.991 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.987+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 >> v1:172.21.15.33:6789/0 conn(0x7fed94071eb0 legacy=0x7fed94146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:30.993 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 10 monclient: _finish_hunting 0 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 1 monclient: found mon.noname-b 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94155d70 con 0x7fed940b1f60 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 10 monclient: _finish_auth 0 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 10 monclient: _check_auth_tickets 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed9bd4d640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed9bd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fed90001690 con 0x7fed940b1f60 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed98ae7640 10 monclient: dump: 2024-03-29T11:44:30.994 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:30.995 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.990+0000 7fed9bd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:30.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed90001860 con 0x7fed940b1f60 2024-03-29T11:44:30.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed98ae7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:30.997 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed9bd4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed9bd4d640 4 set_mon_vals no callback set 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed98ae7640 1 -- 172.21.15.33:0/757750300 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fed90001cb0 con 0x7fed940b1f60 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed98ae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed98ae7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.994+0000 7fed98ae7640 10 monclient: dump: 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:30.998 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:30.999 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.995+0000 7fed9bd4d640 10 monclient: shutdown 2024-03-29T11:44:31.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.995+0000 7fed9bd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:31.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.995+0000 7fed9bd4d640 1 -- 172.21.15.33:0/757750300 >> v1:172.21.15.33:6790/0 conn(0x7fed940b1f60 legacy=0x7fed94143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:31.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.995+0000 7fed9bd4d640 1 -- 172.21.15.33:0/757750300 shutdown_connections 2024-03-29T11:44:31.000 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.995+0000 7fed9bd4d640 1 -- 172.21.15.33:0/757750300 >> 172.21.15.33:0/757750300 conn(0x7fed94066a00 msgr2=0x7fed94066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:31.001 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.998+0000 7fed9bd4d640 1 -- 172.21.15.33:0/757750300 shutdown_connections 2024-03-29T11:44:31.002 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.998+0000 7fed9bd4d640 1 -- 172.21.15.33:0/757750300 wait complete. 2024-03-29T11:44:31.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:30.999+0000 7fed9bd4d640 10 monclient: build_initial_monmap 2024-03-29T11:44:31.003 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 10 monclient: monmap: 2024-03-29T11:44:31.003 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:31.003 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:31.003 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:31.004 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:31.004 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:31.004 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:31.004 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:31.004 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:31.004 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:31.004 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:31.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 1 librados: starting msgr at 2024-03-29T11:44:31.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 1 librados: starting objecter 2024-03-29T11:44:31.006 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 1 Processor -- start 2024-03-29T11:44:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 1 -- start start 2024-03-29T11:44:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 1 librados: setting wanted keys 2024-03-29T11:44:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 1 librados: calling monclient init 2024-03-29T11:44:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.000+0000 7fed9bd4d640 10 monclient: init 2024-03-29T11:44:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.002+0000 7fed9bd4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:31.007 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.002+0000 7fed9bd4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.003+0000 7fed9bd4d640 10 monclient(hunting): picked mon.noname-a con 0x7fed94143d10 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed992e8640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fed94141300 0x7fed94154c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:36932/0 (socket says 172.21.15.33:36932) 2024-03-29T11:44:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed992e8640 1 -- 172.21.15.33:0/3762421207 learned_addr learned my addr 172.21.15.33:0/3762421207 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed9bd4d640 10 monclient(hunting): picked mon.noname-c con 0x7fed94141300 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed9bd4d640 10 monclient(hunting): picked mon.noname-b con 0x7fed9406f620 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed94132740 con 0x7fed94143d10 2024-03-29T11:44:31.009 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed94142230 con 0x7fed9406f620 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed94142510 con 0x7fed94141300 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed9bd4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed9bd4d640 10 monclient(hunting): authenticate will time out at 1427.268241s 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 10 client.?.objecter ms_handle_connect 0x7fed94141300 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170733702 0 0) 0x7fed94142510 con 0x7fed94141300 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:31.010 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.004+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed78004eb0 con 0x7fed94141300 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897853100 0 0) 0x7fed78004eb0 con 0x7fed94141300 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed94142510 con 0x7fed94141300 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fed8c004400 con 0x7fed94141300 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:31.011 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 10 monclient(hunting): dump: 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:31.012 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:31.013 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 609744328 0 0) 0x7fed94142510 con 0x7fed94141300 2024-03-29T11:44:31.013 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.005+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 >> v1:172.21.15.153:6789/0 conn(0x7fed9406f620 legacy=0x7fed94132030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.008+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 >> v1:172.21.15.33:6789/0 conn(0x7fed94143d10 legacy=0x7fed94131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.008+0000 7fed8a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:31.017 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.008+0000 7fed8a7fc640 1 monclient: found mon.c 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed941427f0 con 0x7fed94141300 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 client.?.objecter ms_handle_connect 0x7fed94143d10 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed8c004850 con 0x7fed94141300 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fed8c004660 con 0x7fed94141300 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed8a7fc640 10 monclient: dump: 2024-03-29T11:44:31.018 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.009+0000 7fed9bd4d640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 10 monclient: _renew_subs 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed94142c10 con 0x7fed94141300 2024-03-29T11:44:31.019 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 10 monclient: _renew_subs 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed941e7ec0 con 0x7fed94141300 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed9bd4d640 1 librados: init done 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.010+0000 7fed8bfff640 4 set_mon_vals no callback set 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.012+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fed8c0016a0 con 0x7fed94141300 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed9bd4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed9bd4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7fed8c084de0 con 0x7fed94141300 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed8a7fc640 10 client.4598.objecter ms_dispatch 0x7fed94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed8a7fc640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed8a7fc640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed8a7fc640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed9bd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:31.020 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7c005080 con 0x7fed94141300 2024-03-29T11:44:31.021 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed99ae9640 10 monclient: get_auth_request con 0x7fed7806dbf0 auth_method 0 2024-03-29T11:44:31.021 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.013+0000 7fed8a7fc640 10 client.4598.objecter ms_handle_connect 0x7fed7806dbf0 2024-03-29T11:44:31.025 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.022+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed8c050720 con 0x7fed94141300 2024-03-29T11:44:31.025 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.022+0000 7fed8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:31.025 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.022+0000 7fed8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:31.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.383+0000 7fed9bd4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:31.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.383+0000 7fed9bd4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:31.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.383+0000 7fed9bd4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:31.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.383+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed7c005330 con 0x7fed94141300 2024-03-29T11:44:31.388 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.384+0000 7fed8a7fc640 1 -- 172.21.15.33:0/3762421207 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7fed8c052650 con 0x7fed94141300 2024-03-29T11:44:31.388 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.384+0000 7fed8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:31.388 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.384+0000 7fed8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:31.388 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:31.388 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 10 librados: watch_flush enter 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 10 librados: watch_flush exit 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 >> v1:172.21.15.153:6816/2737028074 conn(0x7fed7806dbf0 legacy=0x7fed780700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 10 monclient: shutdown 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 >> v1:172.21.15.33:6790/0 conn(0x7fed94141300 legacy=0x7fed94154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 shutdown_connections 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 >> 172.21.15.33:0/3762421207 conn(0x7fed94066a00 msgr2=0x7fed94146ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:31.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.388+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 shutdown_connections 2024-03-29T11:44:31.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.389+0000 7fed9bd4d640 1 -- 172.21.15.33:0/3762421207 wait complete. 2024-03-29T11:44:31.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:31.389+0000 7fed9bd4d640 1 librados: shutdown 2024-03-29T11:44:31.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:38.216 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.363+0000 7f8caf1ab640 10 monclient: get_monmap_and_config 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.363+0000 7f8caf1ab640 10 monclient: build_initial_monmap 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.363+0000 7f8caf1ab640 10 monclient: monmap: 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:38.363748+0000 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:38.363748+0000 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:38.367 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:38.368 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:38.368 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:38.368 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.364+0000 7f8caf1ab640 1 Processor -- start 2024-03-29T11:44:38.368 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.364+0000 7f8caf1ab640 1 -- start start 2024-03-29T11:44:38.368 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.364+0000 7f8caf1ab640 10 monclient: init 2024-03-29T11:44:38.369 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.365+0000 7f8caf1ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:38.369 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.365+0000 7f8caf1ab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:38.369 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.366+0000 7f8caf1ab640 10 monclient(hunting): picked mon.noname-c con 0x7f8ca80b4820 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:38.369 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.366+0000 7f8caf1ab640 10 monclient(hunting): picked mon.noname-a con 0x7f8ca80a9ce0 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:38.371 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8caf1ab640 10 monclient(hunting): picked mon.noname-b con 0x7f8ca80b3cd0 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:38.371 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8caf1ab640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca80b4180 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.371 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8caf1ab640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca80aaf60 con 0x7f8ca80b4820 2024-03-29T11:44:38.371 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8caf1ab640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca813c200 con 0x7f8ca80b3cd0 2024-03-29T11:44:38.372 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8caf1ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:38.372 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8caf1ab640 10 monclient(hunting): authenticate will time out at 1434.631296s 2024-03-29T11:44:38.372 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8cae1a9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8ca80b4820 0x7f8ca813baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:34976/0 (socket says 172.21.15.33:34976) 2024-03-29T11:44:38.372 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.367+0000 7f8cae1a9640 1 -- 172.21.15.33:0/568127745 learned_addr learned my addr 172.21.15.33:0/568127745 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:38.373 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.369+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778348475 0 0) 0x7f8ca80aaf60 con 0x7f8ca80b4820 2024-03-29T11:44:38.373 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.369+0000 7f8cad1a7640 10 monclient(hunting): _negotiate 2024-03-29T11:44:38.373 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.369+0000 7f8cad1a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:38.373 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.369+0000 7f8cad1a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:38.373 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.369+0000 7f8cad1a7640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:44:38.373 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.369+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c94003520 con 0x7f8ca80b4820 2024-03-29T11:44:38.380 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.370+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658691803 0 0) 0x7f8c98001110 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.370+0000 7f8cad1a7640 10 monclient(hunting): _negotiate 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.370+0000 7f8cad1a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.370+0000 7f8cad1a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.370+0000 7f8cad1a7640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.370+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca80aaf60 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659981596 0 0) 0x7f8ca813c200 con 0x7f8ca80b3cd0 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient(hunting): _negotiate 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:38.381 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:44:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c940039d0 con 0x7f8ca80b3cd0 2024-03-29T11:44:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2092577171 0 0) 0x7f8ca80aaf60 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 >> v1:172.21.15.33:6790/0 conn(0x7f8ca80b3cd0 legacy=0x7f8ca814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 >> v1:172.21.15.153:6789/0 conn(0x7f8ca80b4820 legacy=0x7f8ca813baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient: _finish_hunting 0 2024-03-29T11:44:38.382 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 monclient: found mon.noname-a 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca8145e80 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient: _finish_auth 0 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient: _check_auth_tickets 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8c980015e0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.383 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.371+0000 7f8cad1a7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.372+0000 7f8cad1a7640 10 monclient: dump: 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:38.384 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.372+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c980017c0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.372+0000 7f8cad1a7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.372+0000 7f8cad1a7640 1 -- 172.21.15.33:0/568127745 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8c980015e0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.372+0000 7f8cad1a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:38.385 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.372+0000 7f8cad1a7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.372+0000 7f8cad1a7640 10 monclient: dump: 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:38.386 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.373+0000 7f8caf1ab640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.373+0000 7f8caf1ab640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.373+0000 7f8caf1ab640 4 set_mon_vals no callback set 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.374+0000 7f8caf1ab640 10 monclient: shutdown 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.374+0000 7f8caf1ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.374+0000 7f8caf1ab640 1 -- 172.21.15.33:0/568127745 >> v1:172.21.15.33:6789/0 conn(0x7f8ca80a9ce0 legacy=0x7f8ca8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.374+0000 7f8caf1ab640 1 -- 172.21.15.33:0/568127745 shutdown_connections 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.374+0000 7f8caf1ab640 1 -- 172.21.15.33:0/568127745 >> 172.21.15.33:0/568127745 conn(0x7f8ca806d070 msgr2=0x7f8ca8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.375+0000 7f8caf1ab640 1 -- 172.21.15.33:0/568127745 shutdown_connections 2024-03-29T11:44:38.387 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.375+0000 7f8caf1ab640 1 -- 172.21.15.33:0/568127745 wait complete. 2024-03-29T11:44:38.392 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 10 monclient: build_initial_monmap 2024-03-29T11:44:38.393 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 10 monclient: monmap: 2024-03-29T11:44:38.393 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:38.393 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:38.393 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:38.393 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:38.393 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:38.394 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:38.394 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:38.394 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:38.394 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:38.394 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:38.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 1 librados: starting msgr at 2024-03-29T11:44:38.394 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 1 librados: starting objecter 2024-03-29T11:44:38.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 1 Processor -- start 2024-03-29T11:44:38.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 1 -- start start 2024-03-29T11:44:38.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 1 librados: setting wanted keys 2024-03-29T11:44:38.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 1 librados: calling monclient init 2024-03-29T11:44:38.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.376+0000 7f8caf1ab640 10 monclient: init 2024-03-29T11:44:38.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.377+0000 7f8caf1ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:38.395 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.377+0000 7f8caf1ab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 10 monclient(hunting): picked mon.noname-c con 0x7f8ca80a9ce0 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 10 monclient(hunting): picked mon.noname-b con 0x7f8ca80b3cd0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 10 monclient(hunting): picked mon.noname-a con 0x7f8ca80b4820 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca806e450 con 0x7f8ca80b4820 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca813efd0 con 0x7f8ca80b3cd0 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca813eab0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:38.396 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.378+0000 7f8caf1ab640 10 monclient(hunting): authenticate will time out at 1434.642262s 2024-03-29T11:44:38.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.380+0000 7f8cae1a9640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f8ca80a9ce0 0x7f8ca813e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:53855/0 (socket says 172.21.15.33:53855) 2024-03-29T11:44:38.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.380+0000 7f8cae1a9640 1 -- 172.21.15.33:0/3441457700 learned_addr learned my addr 172.21.15.33:0/3441457700 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:38.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.380+0000 7f8c9effd640 10 client.?.objecter ms_handle_connect 0x7f8ca80a9ce0 2024-03-29T11:44:38.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.380+0000 7f8c9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:38.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275396606 0 0) 0x7f8ca813eab0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:38.397 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:38.398 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:38.398 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:44:38.398 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c004eb0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.398 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 client.?.objecter ms_handle_connect 0x7f8ca80b3cd0 2024-03-29T11:44:38.398 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:38.398 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842003818 0 0) 0x7f8ca813efd0 con 0x7f8ca80b3cd0 2024-03-29T11:44:38.398 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:44:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca813eab0 con 0x7f8ca80b3cd0 2024-03-29T11:44:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 client.?.objecter ms_handle_connect 0x7f8ca80b4820 2024-03-29T11:44:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:38.399 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1489585140 0 0) 0x7f8c8c004eb0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ca813efd0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8ca4001830 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): dump: 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:38.400 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:38.401 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009758825 0 0) 0x7f8ca806e450 con 0x7f8ca80b4820 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c004eb0 con 0x7f8ca80b4820 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.381+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 671592928 0 0) 0x7f8ca813efd0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.402 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 >> v1:172.21.15.153:6789/0 conn(0x7f8ca80b3cd0 legacy=0x7f8ca8133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 >> v1:172.21.15.33:6789/0 conn(0x7f8ca80b4820 legacy=0x7f8ca81e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 1 monclient: found mon.c 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca813d9e0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: _finish_auth 0 2024-03-29T11:44:38.403 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 10 monclient: _renew_subs 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca81441b0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 10 monclient: _renew_subs 2024-03-29T11:44:38.404 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:38.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca8134540 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8caf1ab640 1 librados: init done 2024-03-29T11:44:38.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ca4001ce0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:38.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f8ca4004980 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:38.405 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:38.406 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8c9effd640 10 monclient: dump: 2024-03-29T11:44:38.406 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:38.406 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:38.406 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.406 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:38.406 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:38.406 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:38.407 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:38.407 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:38.407 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:38.407 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.382+0000 7f8cac9a6640 4 set_mon_vals no callback set 2024-03-29T11:44:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.383+0000 7f8caf1ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:38.407 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.383+0000 7f8caf1ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.383+0000 7f8caf1ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.383+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c90005080 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.384+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f8ca4004ae0 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.385+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1662772502 0 0) 0x7f8ca4084a00 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.385+0000 7f8c9effd640 10 client.4604.objecter ms_dispatch 0x7f8ca8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.385+0000 7f8c9effd640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:38.408 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.385+0000 7f8c9effd640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.385+0000 7f8c9effd640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:44:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.385+0000 7f8cad9a8640 10 monclient: get_auth_request con 0x7f8c8c06de20 auth_method 0 2024-03-29T11:44:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.386+0000 7f8c9effd640 10 client.4604.objecter ms_handle_connect 0x7f8c8c06de20 2024-03-29T11:44:38.409 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.393+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ca4050110 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.393+0000 7f8c9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:38.410 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.393+0000 7f8c9effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:38.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.728+0000 7f8caf1ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:38.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.728+0000 7f8caf1ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:38.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.728+0000 7f8caf1ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:44:38.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.728+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c90005330 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.736 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.729+0000 7f8c9effd640 1 -- 172.21.15.33:0/3441457700 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2358064096) 0x7f8ca4052210 con 0x7f8ca80a9ce0 2024-03-29T11:44:38.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.729+0000 7f8c9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:38.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.729+0000 7f8c9effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:38.737 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:38.737 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":10,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:34:40.847501+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:40.847501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.734+0000 7f8caf1ab640 10 librados: watch_flush enter 2024-03-29T11:44:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.734+0000 7f8caf1ab640 10 librados: watch_flush exit 2024-03-29T11:44:38.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.734+0000 7f8caf1ab640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:44:38.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.734+0000 7f8caf1ab640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:44:38.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 >> v1:172.21.15.153:6816/2737028074 conn(0x7f8c8c06de20 legacy=0x7f8c8c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 10 monclient: shutdown 2024-03-29T11:44:38.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:38.742 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 >> v1:172.21.15.33:6790/0 conn(0x7f8ca80a9ce0 legacy=0x7f8ca813e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:38.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 shutdown_connections 2024-03-29T11:44:38.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 >> 172.21.15.33:0/3441457700 conn(0x7f8ca806d070 msgr2=0x7f8ca814e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:38.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 shutdown_connections 2024-03-29T11:44:38.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 1 -- 172.21.15.33:0/3441457700 wait complete. 2024-03-29T11:44:38.743 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:38.737+0000 7f8caf1ab640 1 librados: shutdown 2024-03-29T11:44:38.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:45.566 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.710+0000 7f482d13d640 10 monclient: get_monmap_and_config 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.710+0000 7f482d13d640 10 monclient: build_initial_monmap 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.710+0000 7f482d13d640 10 monclient: monmap: 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:45.710766+0000 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:45.710766+0000 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:45.714 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.711+0000 7f482d13d640 1 Processor -- start 2024-03-29T11:44:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.711+0000 7f482d13d640 1 -- start start 2024-03-29T11:44:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.711+0000 7f482d13d640 10 monclient: init 2024-03-29T11:44:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.711+0000 7f482d13d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:45.715 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 10 monclient(hunting): picked mon.noname-c con 0x7f48280b01a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 10 monclient(hunting): picked mon.noname-b con 0x7f48280af010 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 10 monclient(hunting): picked mon.noname-a con 0x7f48280ac5a0 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48280ae8a0 con 0x7f48280ac5a0 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48280ab1d0 con 0x7f48280b01a0 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48280ab4b0 con 0x7f48280af010 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f482d13d640 10 monclient(hunting): authenticate will time out at 1441.975797s 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f4826d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f48280b01a0 0x7f48280abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:43680/0 (socket says 172.21.15.33:43680) 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f4826d9d640 1 -- 172.21.15.33:0/4137530658 learned_addr learned my addr 172.21.15.33:0/4137530658 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.712+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332196789 0 0) 0x7f48280ab1d0 con 0x7f48280b01a0 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.713+0000 7f4825d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.713+0000 7f4825d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:45.717 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.713+0000 7f4825d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.713+0000 7f4825d9b640 10 monclient(hunting): my global_id is 4603 2024-03-29T11:44:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.713+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4814003520 con 0x7f48280b01a0 2024-03-29T11:44:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.714+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2988927408 0 0) 0x7f4814003520 con 0x7f48280b01a0 2024-03-29T11:44:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.714+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 >> v1:172.21.15.33:6790/0 conn(0x7f48280af010 legacy=0x7f4828149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:45.718 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.715+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 >> v1:172.21.15.33:6789/0 conn(0x7f48280ac5a0 legacy=0x7f4828151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 1 monclient: found mon.noname-c 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48280ab790 con 0x7f48280b01a0 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 10 monclient: _finish_auth 0 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:45.722 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4818001a10 con 0x7f48280b01a0 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.716+0000 7f4825d9b640 10 monclient: dump: 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f482d13d640 5 monclient: authenticate success, global_id 4603 2024-03-29T11:44:45.723 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f482d13d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4818001ca0 con 0x7f48280b01a0 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f4825d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f4825d9b640 1 -- 172.21.15.33:0/4137530658 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4818002030 con 0x7f48280b01a0 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f482d13d640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f482d13d640 4 set_mon_vals no callback set 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f4825d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f4825d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.717+0000 7f4825d9b640 10 monclient: dump: 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:45.724 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:45.725 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:45.725 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:45.725 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.718+0000 7f482d13d640 10 monclient: shutdown 2024-03-29T11:44:45.725 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.718+0000 7f482d13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:45.725 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.718+0000 7f482d13d640 1 -- 172.21.15.33:0/4137530658 >> v1:172.21.15.153:6789/0 conn(0x7f48280b01a0 legacy=0x7f48280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:45.725 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.718+0000 7f482d13d640 1 -- 172.21.15.33:0/4137530658 shutdown_connections 2024-03-29T11:44:45.725 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.718+0000 7f482d13d640 1 -- 172.21.15.33:0/4137530658 >> 172.21.15.33:0/4137530658 conn(0x7f482806b660 msgr2=0x7f482806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.719+0000 7f482d13d640 1 -- 172.21.15.33:0/4137530658 shutdown_connections 2024-03-29T11:44:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.719+0000 7f482d13d640 1 -- 172.21.15.33:0/4137530658 wait complete. 2024-03-29T11:44:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.720+0000 7f482d13d640 10 monclient: build_initial_monmap 2024-03-29T11:44:45.726 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.720+0000 7f482d13d640 10 monclient: monmap: 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.721+0000 7f482d13d640 1 librados: starting msgr at 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.721+0000 7f482d13d640 1 librados: starting objecter 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 1 Processor -- start 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 1 -- start start 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 1 librados: setting wanted keys 2024-03-29T11:44:45.727 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 1 librados: calling monclient init 2024-03-29T11:44:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 10 monclient: init 2024-03-29T11:44:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 10 monclient(hunting): picked mon.noname-b con 0x7f48280ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:45.728 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.722+0000 7f482d13d640 10 monclient(hunting): picked mon.noname-c con 0x7f48280b01a0 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f4826d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f48280ac5a0 0x7f4828125b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:44060/0 (socket says 172.21.15.33:44060) 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f482d13d640 10 monclient(hunting): picked mon.noname-a con 0x7f48281df460 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f482d13d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48281277d0 con 0x7f48281df460 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f482d13d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48281263d0 con 0x7f48280ac5a0 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f482d13d640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4828125370 con 0x7f48280b01a0 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f482d13d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f482d13d640 10 monclient(hunting): authenticate will time out at 1441.990296s 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f4826d9d640 1 -- 172.21.15.33:0/1424000424 learned_addr learned my addr 172.21.15.33:0/1424000424 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f481f7fe640 10 client.?.objecter ms_handle_connect 0x7f48281df460 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.726+0000 7f481f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:45.732 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659353471 0 0) 0x7f48281277d0 con 0x7f48281df460 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4808004eb0 con 0x7f48281df460 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3416601624 0 0) 0x7f4808004eb0 con 0x7f48281df460 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48281277d0 con 0x7f48281df460 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f48100017e0 con 0x7f48281df460 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.727+0000 7f481f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.728+0000 7f481f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.728+0000 7f481f7fe640 10 monclient(hunting): dump: 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.733 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.728+0000 7f481f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.728+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3546082673 0 0) 0x7f48281277d0 con 0x7f48281df460 2024-03-29T11:44:45.734 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.728+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 >> v1:172.21.15.33:6790/0 conn(0x7f48280b01a0 legacy=0x7f48281ded50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.730+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 >> v1:172.21.15.153:6789/0 conn(0x7f48280ac5a0 legacy=0x7f4828125b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.730+0000 7f481f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 1 monclient: found mon.a 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48281268f0 con 0x7f48281df460 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:45.737 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 client.?.objecter ms_handle_connect 0x7f48280ac5a0 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4810001a40 con 0x7f48281df460 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f4810003850 con 0x7f48281df460 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 10 monclient: _renew_subs 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4828070450 con 0x7f48281df460 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f481f7fe640 10 monclient: dump: 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:45.738 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 10 monclient: _renew_subs 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4828144a10 con 0x7f48281df460 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.731+0000 7f482d13d640 1 librados: init done 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.732+0000 7f482559a640 4 set_mon_vals no callback set 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.732+0000 7f482d13d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:45.739 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.732+0000 7f482d13d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:45.740 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.732+0000 7f482d13d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:45.740 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.732+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480c005080 con 0x7f48281df460 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.734+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f4810003a30 con 0x7f48281df460 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.735+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7f4810083950 con 0x7f48281df460 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.735+0000 7f481f7fe640 10 client.4584.objecter ms_dispatch 0x7f4828000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.735+0000 7f481f7fe640 3 client.4584.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.735+0000 7f481f7fe640 3 client.4584.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.735+0000 7f4826d9d640 10 monclient: get_auth_request con 0x7f480806dac0 auth_method 0 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.735+0000 7f481f7fe640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:44:45.741 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.735+0000 7f481f7fe640 10 client.4584.objecter ms_handle_connect 0x7f480806dac0 2024-03-29T11:44:45.746 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.743+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f481004f060 con 0x7f48281df460 2024-03-29T11:44:45.746 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.743+0000 7f481f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:45.746 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:45.743+0000 7f481f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:46.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.070+0000 7f482d13d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:46.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.070+0000 7f482d13d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:46.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.070+0000 7f482d13d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.33:6789/0 2024-03-29T11:44:46.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.070+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f480c005330 con 0x7f48281df460 2024-03-29T11:44:46.076 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:46.077 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:46.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.072+0000 7f481f7fe640 1 -- 172.21.15.33:0/1424000424 <== mon.0 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7f4810051160 con 0x7f48281df460 2024-03-29T11:44:46.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.072+0000 7f481f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:46.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.072+0000 7f481f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:46.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.076+0000 7f482d13d640 10 librados: watch_flush enter 2024-03-29T11:44:46.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.076+0000 7f482d13d640 10 librados: watch_flush exit 2024-03-29T11:44:46.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.076+0000 7f482d13d640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.076+0000 7f482d13d640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.076+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 >> v1:172.21.15.153:6816/2737028074 conn(0x7f480806dac0 legacy=0x7f480806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.077+0000 7f482d13d640 10 monclient: shutdown 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.077+0000 7f482d13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.077+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 >> v1:172.21.15.33:6789/0 conn(0x7f48281df460 legacy=0x7f4828144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.077+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 shutdown_connections 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.077+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 >> 172.21.15.33:0/1424000424 conn(0x7f482806b660 msgr2=0x7f482806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:46.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.077+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 shutdown_connections 2024-03-29T11:44:46.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.077+0000 7f482d13d640 1 -- 172.21.15.33:0/1424000424 wait complete. 2024-03-29T11:44:46.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:46.078+0000 7f482d13d640 1 librados: shutdown 2024-03-29T11:44:46.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:52.908 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.052+0000 7f587a65f640 10 monclient: get_monmap_and_config 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.052+0000 7f587a65f640 10 monclient: build_initial_monmap 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.052+0000 7f587a65f640 10 monclient: monmap: 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:44:53.052774+0000 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:44:53.052774+0000 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:53.056 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:53.057 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:44:53.057 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:44:53.057 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:53.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.053+0000 7f587a65f640 1 Processor -- start 2024-03-29T11:44:53.057 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.053+0000 7f587a65f640 1 -- start start 2024-03-29T11:44:53.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.053+0000 7f587a65f640 10 monclient: init 2024-03-29T11:44:53.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:53.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:53.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 10 monclient(hunting): picked mon.noname-c con 0x7f586c006eb0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:53.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 10 monclient(hunting): picked mon.noname-a con 0x7f586c038f10 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:53.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 10 monclient(hunting): picked mon.noname-b con 0x7f586c036690 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:53.058 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c033c90 con 0x7f586c038f10 2024-03-29T11:44:53.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c0389f0 con 0x7f586c006eb0 2024-03-29T11:44:53.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c030890 con 0x7f586c036690 2024-03-29T11:44:53.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:53.059 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.054+0000 7f587a65f640 10 monclient(hunting): authenticate will time out at 1449.318231s 2024-03-29T11:44:53.062 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.058+0000 7f5878e5c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f586c038f10 0x7f586c039330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:52723/0 (socket says 172.21.15.33:52723) 2024-03-29T11:44:53.062 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.058+0000 7f5878e5c640 1 -- 172.21.15.33:0/777886208 learned_addr learned my addr 172.21.15.33:0/777886208 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:53.062 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856678341 0 0) 0x7f586c033c90 con 0x7f586c038f10 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5860003520 con 0x7f586c038f10 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 <== mon.0 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965887574 0 0) 0x7f5860003520 con 0x7f586c038f10 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 >> v1:172.21.15.33:6790/0 conn(0x7f586c036690 legacy=0x7f586c036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 >> v1:172.21.15.153:6789/0 conn(0x7f586c006eb0 legacy=0x7f586c031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 monclient: found mon.noname-a 2024-03-29T11:44:53.066 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.33:6789/0 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c0cbc40 con 0x7f586c038f10 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: _finish_auth 0 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 <== mon.0 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5864001690 con 0x7f586c038f10 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: dump: 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:53.067 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 <== mon.0 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5864001920 con 0x7f586c038f10 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 1 -- 172.21.15.33:0/777886208 <== mon.0 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5864001690 con 0x7f586c038f10 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.059+0000 7f586bfff640 10 monclient: dump: 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:53.068 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.060+0000 7f587a65f640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.060+0000 7f587a65f640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.060+0000 7f587a65f640 4 set_mon_vals no callback set 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.061+0000 7f587a65f640 10 monclient: shutdown 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.061+0000 7f587a65f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.061+0000 7f587a65f640 1 -- 172.21.15.33:0/777886208 >> v1:172.21.15.33:6789/0 conn(0x7f586c038f10 legacy=0x7f586c039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.061+0000 7f587a65f640 1 -- 172.21.15.33:0/777886208 shutdown_connections 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.061+0000 7f587a65f640 1 -- 172.21.15.33:0/777886208 >> 172.21.15.33:0/777886208 conn(0x7f586c005c20 msgr2=0x7f586c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:53.069 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.062+0000 7f587a65f640 1 -- 172.21.15.33:0/777886208 shutdown_connections 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.062+0000 7f587a65f640 1 -- 172.21.15.33:0/777886208 wait complete. 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 10 monclient: build_initial_monmap 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 10 monclient: monmap: 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:44:53.070 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 1 librados: starting msgr at 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 1 librados: starting objecter 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 1 Processor -- start 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 1 -- start start 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 1 librados: setting wanted keys 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 1 librados: calling monclient init 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.063+0000 7f587a65f640 10 monclient: init 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.064+0000 7f587a65f640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.064+0000 7f587a65f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.064+0000 7f587a65f640 10 monclient(hunting): picked mon.noname-b con 0x7f586c006eb0 addr v1:172.21.15.153:6789/0 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.064+0000 7f587a65f640 10 monclient(hunting): picked mon.noname-c con 0x7f586c036690 addr v1:172.21.15.33:6790/0 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.064+0000 7f587a65f640 10 monclient(hunting): picked mon.noname-a con 0x7f586c0c91c0 addr v1:172.21.15.33:6789/0 2024-03-29T11:44:53.071 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.064+0000 7f587a65f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c0c8d30 con 0x7f586c0c91c0 2024-03-29T11:44:53.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.067+0000 7f5879e5e640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f586c0c91c0 0x7f586c185910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:52845/0 (socket says 172.21.15.33:52845) 2024-03-29T11:44:53.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.067+0000 7f5879e5e640 1 -- 172.21.15.33:0/2992200339 learned_addr learned my addr 172.21.15.33:0/2992200339 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:44:53.072 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.067+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c00b5e0 con 0x7f586c006eb0 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.067+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c00b8c0 con 0x7f586c036690 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.067+0000 7f587a65f640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.067+0000 7f587a65f640 10 monclient(hunting): authenticate will time out at 1449.331575s 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 10 client.?.objecter ms_handle_connect 0x7f586c006eb0 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179582844 0 0) 0x7f586c00b5e0 con 0x7f586c006eb0 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:53.073 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.069+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5858004eb0 con 0x7f586c006eb0 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 10 client.?.objecter ms_handle_connect 0x7f586c036690 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 245610312 0 0) 0x7f5858004eb0 con 0x7f586c006eb0 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f586c00b5e0 con 0x7f586c006eb0 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5870001930 con 0x7f586c006eb0 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 10 monclient(hunting): dump: 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.074 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.075 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:53.075 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:53.075 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:53.075 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:53.075 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:53.075 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:53.075 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.070+0000 7f5869ffb640 10 client.?.objecter ms_handle_connect 0x7f586c0c91c0 2024-03-29T11:44:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.071+0000 7f5869ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.071+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496030461 0 0) 0x7f586c00b8c0 con 0x7f586c036690 2024-03-29T11:44:53.076 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.071+0000 7f5869ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.071+0000 7f5869ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.071+0000 7f5869ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.071+0000 7f5869ffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.071+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5858004eb0 con 0x7f586c036690 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1808149868 0 0) 0x7f586c00b5e0 con 0x7f586c006eb0 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 >> v1:172.21.15.33:6790/0 conn(0x7f586c036690 legacy=0x7f586c0097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 >> v1:172.21.15.33:6789/0 conn(0x7f586c0c91c0 legacy=0x7f586c185910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:53.077 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 1 monclient: found mon.b 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c00a6c0 con 0x7f586c006eb0 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.072+0000 7f5869ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 5 monclient: authenticate success, global_id 4609 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 10 monclient: _renew_subs 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586c009ec0 con 0x7f586c006eb0 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 10 monclient: _renew_subs 2024-03-29T11:44:53.078 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c0c53a0 con 0x7f586c006eb0 2024-03-29T11:44:53.079 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.073+0000 7f587a65f640 1 librados: init done 2024-03-29T11:44:53.080 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.075+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5870001a90 con 0x7f586c006eb0 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.075+0000 7f5869ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.075+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5870003a60 con 0x7f586c006eb0 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.075+0000 7f5869ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.075+0000 7f586b7fe640 4 set_mon_vals no callback set 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.075+0000 7f5869ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.076+0000 7f5869ffb640 10 monclient: dump: 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:44:53.081 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:44:53.082 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:44:53.082 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:44:53.082 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:44:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.076+0000 7f587a65f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.076+0000 7f587a65f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.076+0000 7f587a65f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:53.082 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.077+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5854005080 con 0x7f586c006eb0 2024-03-29T11:44:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.077+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f5870003bc0 con 0x7f586c006eb0 2024-03-29T11:44:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.078+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7f58700838e0 con 0x7f586c006eb0 2024-03-29T11:44:53.083 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.078+0000 7f5869ffb640 10 client.4609.objecter ms_dispatch 0x7f5874000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:53.084 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.078+0000 7f5869ffb640 3 client.4609.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:53.084 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.078+0000 7f5869ffb640 3 client.4609.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:53.084 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.078+0000 7f5869ffb640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-29T11:44:53.084 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.079+0000 7f5878e5c640 10 monclient: get_auth_request con 0x7f585806dcd0 auth_method 0 2024-03-29T11:44:53.084 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.079+0000 7f5869ffb640 10 client.4609.objecter ms_handle_connect 0x7f585806dcd0 2024-03-29T11:44:53.089 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.086+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f587004eff0 con 0x7f586c006eb0 2024-03-29T11:44:53.090 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.086+0000 7f5869ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:53.090 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.086+0000 7f5869ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:53.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.421+0000 7f587a65f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:53.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.422+0000 7f587a65f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:53.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.422+0000 7f587a65f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:44:53.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.422+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5854005330 con 0x7f586c006eb0 2024-03-29T11:44:53.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.423+0000 7f5869ffb640 1 -- 172.21.15.33:0/2992200339 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7f58700510f0 con 0x7f586c006eb0 2024-03-29T11:44:53.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.423+0000 7f5869ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:53.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.423+0000 7f5869ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:53.427 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:44:53.427 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 10 librados: watch_flush enter 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 10 librados: watch_flush exit 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 10 client.4609.objecter successfully canceled tick 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 >> v1:172.21.15.153:6816/2737028074 conn(0x7f585806dcd0 legacy=0x7f5858070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 10 monclient: shutdown 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:53.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.426+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 >> v1:172.21.15.153:6789/0 conn(0x7f586c006eb0 legacy=0x7f586c0cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:53.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.428+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 shutdown_connections 2024-03-29T11:44:53.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.428+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 >> 172.21.15.33:0/2992200339 conn(0x7f586c005c20 msgr2=0x7f586c03c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:53.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.429+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 shutdown_connections 2024-03-29T11:44:53.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.429+0000 7f587a65f640 1 -- 172.21.15.33:0/2992200339 wait complete. 2024-03-29T11:44:53.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:44:53.429+0000 7f587a65f640 1 librados: shutdown 2024-03-29T11:44:53.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:00.258 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.408+0000 7f971d806640 10 monclient: get_monmap_and_config 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.408+0000 7f971d806640 10 monclient: build_initial_monmap 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.408+0000 7f971d806640 10 monclient: monmap: 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:45:00.408773+0000 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:45:00.408773+0000 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:00.412 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:45:00.413 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:45:00.413 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:00.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.409+0000 7f971d806640 1 Processor -- start 2024-03-29T11:45:00.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.410+0000 7f971d806640 1 -- start start 2024-03-29T11:45:00.413 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.410+0000 7f971d806640 10 monclient: init 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 10 monclient(hunting): picked mon.noname-c con 0x7f97180b4890 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 10 monclient(hunting): picked mon.noname-a con 0x7f971813c040 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 10 monclient(hunting): picked mon.noname-b con 0x7f97181427e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97180b42f0 con 0x7f971813c040 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f97167fc640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f971813c040 0x7f97181420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:59606/0 (socket says 172.21.15.33:59606) 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f97167fc640 1 -- 172.21.15.33:0/758397479 learned_addr learned my addr 172.21.15.33:0/758397479 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 1 -- 172.21.15.33:0/758397479 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97180aa7a0 con 0x7f97180b4890 2024-03-29T11:45:00.415 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 1 -- 172.21.15.33:0/758397479 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97180aaa10 con 0x7f97181427e0 2024-03-29T11:45:00.416 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:00.416 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.411+0000 7f971d806640 10 monclient(hunting): authenticate will time out at 1456.675176s 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348084971 0 0) 0x7f97180aa7a0 con 0x7f97180b4890 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9700003520 con 0x7f97180b4890 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973127067 0 0) 0x7f9700003520 con 0x7f97180b4890 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 >> v1:172.21.15.33:6790/0 conn(0x7f97181427e0 legacy=0x7f971814d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 >> v1:172.21.15.33:6789/0 conn(0x7f971813c040 legacy=0x7f97181420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 monclient: found mon.noname-c 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:00.418 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97180aacf0 con 0x7f97180b4890 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9710001a10 con 0x7f97180b4890 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f9715ffb640 10 monclient: dump: 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:00.419 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f971d806640 5 monclient: authenticate success, global_id 4612 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.412+0000 7f971d806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9710001b70 con 0x7f97180b4890 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f9715ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f9715ffb640 1 -- 172.21.15.33:0/758397479 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9710001f00 con 0x7f97180b4890 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f971d806640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f971d806640 4 set_mon_vals no callback set 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f9715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f9715ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f9715ffb640 10 monclient: dump: 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:00.420 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f971d806640 10 monclient: shutdown 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f971d806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.413+0000 7f971d806640 1 -- 172.21.15.33:0/758397479 >> v1:172.21.15.153:6789/0 conn(0x7f97180b4890 legacy=0x7f971813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.414+0000 7f971d806640 1 -- 172.21.15.33:0/758397479 shutdown_connections 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.414+0000 7f971d806640 1 -- 172.21.15.33:0/758397479 >> 172.21.15.33:0/758397479 conn(0x7f971806d180 msgr2=0x7f9718066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.414+0000 7f971d806640 1 -- 172.21.15.33:0/758397479 shutdown_connections 2024-03-29T11:45:00.421 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.414+0000 7f971d806640 1 -- 172.21.15.33:0/758397479 wait complete. 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.415+0000 7f971d806640 10 monclient: build_initial_monmap 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.415+0000 7f971d806640 10 monclient: monmap: 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.415+0000 7f971d806640 1 librados: starting msgr at 2024-03-29T11:45:00.424 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.415+0000 7f971d806640 1 librados: starting objecter 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.416+0000 7f971d806640 1 Processor -- start 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.416+0000 7f971d806640 1 -- start start 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.416+0000 7f971d806640 1 librados: setting wanted keys 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.416+0000 7f971d806640 1 librados: calling monclient init 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.416+0000 7f971d806640 10 monclient: init 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f971d806640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f971d806640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f971d806640 10 monclient(hunting): picked mon.noname-a con 0x7f97180b4890 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f971d806640 10 monclient(hunting): picked mon.noname-b con 0x7f971813c040 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f97167fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f971813c040 0x7f9718137cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:39836/0 (socket says 172.21.15.33:39836) 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f97167fc640 1 -- 172.21.15.33:0/2766908769 learned_addr learned my addr 172.21.15.33:0/2766908769 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f971d806640 10 monclient(hunting): picked mon.noname-c con 0x7f97181427e0 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97181da370 con 0x7f97180b4890 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97181da650 con 0x7f971813c040 2024-03-29T11:45:00.425 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f96fffff640 10 client.?.objecter ms_handle_connect 0x7f971813c040 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.417+0000 7f96fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.418+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97181db1f0 con 0x7f97181427e0 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.418+0000 7f971d806640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.418+0000 7f971d806640 10 monclient(hunting): authenticate will time out at 1456.681884s 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.418+0000 7f96fffff640 10 client.?.objecter ms_handle_connect 0x7f97180b4890 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.418+0000 7f96fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 client.?.objecter ms_handle_connect 0x7f97181427e0 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464231576 0 0) 0x7f97181da370 con 0x7f97180b4890 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9704004eb0 con 0x7f97180b4890 2024-03-29T11:45:00.426 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694025225 0 0) 0x7f97181da650 con 0x7f971813c040 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): my global_id is 4615 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97181da370 con 0x7f971813c040 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614419576 0 0) 0x7f97181db1f0 con 0x7f97181427e0 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97181da650 con 0x7f97181427e0 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2168501800 0 0) 0x7f97181da370 con 0x7f971813c040 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97181db1f0 con 0x7f971813c040 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9708001870 con 0x7f971813c040 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:00.427 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): dump: 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.419+0000 7f96fffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1436804817 0 0) 0x7f97181db1f0 con 0x7f971813c040 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 >> v1:172.21.15.33:6790/0 conn(0x7f97181427e0 legacy=0x7f97181e2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.428 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 >> v1:172.21.15.33:6789/0 conn(0x7f97180b4890 legacy=0x7f97181daae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 1 monclient: found mon.b 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97180acad0 con 0x7f971813c040 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 10 monclient: _finish_auth 0 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:00.429 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:00.430 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9708001ca0 con 0x7f971813c040 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f971d806640 5 monclient: authenticate success, global_id 4615 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f971d806640 10 monclient: _renew_subs 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f971d806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971813ecf0 con 0x7f971813c040 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f97157fa640 4 set_mon_vals no callback set 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.420+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f9708004980 con 0x7f971813c040 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f96fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f96fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f96fffff640 10 monclient: dump: 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.431 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f971d806640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f971d806640 10 monclient: _renew_subs 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f971d806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f971813f740 con 0x7f971813c040 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.421+0000 7f971d806640 1 librados: init done 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.422+0000 7f971d806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.422+0000 7f971d806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.422+0000 7f971d806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.422+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971804f730 con 0x7f971813c040 2024-03-29T11:45:00.432 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.423+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f9708004ae0 con 0x7f971813c040 2024-03-29T11:45:00.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.424+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7f9708084a40 con 0x7f971813c040 2024-03-29T11:45:00.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.424+0000 7f96fffff640 10 client.4615.objecter ms_dispatch 0x7f9718000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:00.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.424+0000 7f9716ffd640 10 monclient: get_auth_request con 0x7f970406db70 auth_method 0 2024-03-29T11:45:00.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.424+0000 7f96fffff640 3 client.4615.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:00.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.424+0000 7f96fffff640 3 client.4615.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:00.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.424+0000 7f96fffff640 20 client.4615.objecter dump_active .. 0 homeless 2024-03-29T11:45:00.433 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.424+0000 7f96fffff640 10 client.4615.objecter ms_handle_connect 0x7f970406db70 2024-03-29T11:45:00.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.430+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9708050150 con 0x7f971813c040 2024-03-29T11:45:00.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.431+0000 7f96fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:00.434 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.431+0000 7f96fffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:00.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.828+0000 7f971d806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:00.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.828+0000 7f971d806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:00.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.828+0000 7f971d806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:00.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.828+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97181da800 con 0x7f971813c040 2024-03-29T11:45:00.833 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.829+0000 7f96fffff640 1 -- 172.21.15.33:0/2766908769 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7f97181da800 con 0x7f971813c040 2024-03-29T11:45:00.833 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.829+0000 7f96fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:00.833 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.829+0000 7f96fffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:00.833 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:45:00.834 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.833+0000 7f971d806640 10 librados: watch_flush enter 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.833+0000 7f971d806640 10 librados: watch_flush exit 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 20 client.4615.objecter shutdown clearing up homeless session... 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 10 client.4615.objecter successfully canceled tick 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 >> v1:172.21.15.153:6816/2737028074 conn(0x7f970406db70 legacy=0x7f9704070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 10 monclient: shutdown 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 >> v1:172.21.15.153:6789/0 conn(0x7f971813c040 legacy=0x7f9718137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 shutdown_connections 2024-03-29T11:45:00.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 >> 172.21.15.33:0/2766908769 conn(0x7f971806d180 msgr2=0x7f97180b8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:00.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 shutdown_connections 2024-03-29T11:45:00.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 1 -- 172.21.15.33:0/2766908769 wait complete. 2024-03-29T11:45:00.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:00.834+0000 7f971d806640 1 librados: shutdown 2024-03-29T11:45:00.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:07.661 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:07.811 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.805+0000 7f45d4d32640 10 monclient: get_monmap_and_config 2024-03-29T11:45:07.811 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.805+0000 7f45d4d32640 10 monclient: build_initial_monmap 2024-03-29T11:45:07.811 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.805+0000 7f45d4d32640 10 monclient: monmap: 2024-03-29T11:45:07.811 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:07.811 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:45:07.807480+0000 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:45:07.807480+0000 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:45:07.812 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:07.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.806+0000 7f45d4d32640 1 Processor -- start 2024-03-29T11:45:07.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.807+0000 7f45d4d32640 1 -- start start 2024-03-29T11:45:07.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.807+0000 7f45d4d32640 10 monclient: init 2024-03-29T11:45:07.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:07.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:07.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 10 monclient(hunting): picked mon.noname-a con 0x7f45d01457f0 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:07.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 10 monclient(hunting): picked mon.noname-c con 0x7f45d013f650 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:07.816 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 10 monclient(hunting): picked mon.noname-b con 0x7f45d00a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:07.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00aaef0 con 0x7f45d01457f0 2024-03-29T11:45:07.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d0145bd0 con 0x7f45d013f650 2024-03-29T11:45:07.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d0145eb0 con 0x7f45d00a9d70 2024-03-29T11:45:07.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:07.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.808+0000 7f45d4d32640 10 monclient(hunting): authenticate will time out at 1464.073092s 2024-03-29T11:45:07.819 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.814+0000 7f45cdd9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f45d013f650 0x7f45d00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:44477/0 (socket says 172.21.15.33:44477) 2024-03-29T11:45:07.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.814+0000 7f45cdd9b640 1 -- 172.21.15.33:0/1977673548 learned_addr learned my addr 172.21.15.33:0/1977673548 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:07.820 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.815+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899258266 0 0) 0x7f45d0145bd0 con 0x7f45d013f650 2024-03-29T11:45:07.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.815+0000 7f45cd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.815+0000 7f45cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.815+0000 7f45cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.815+0000 7f45cd59a640 10 monclient(hunting): my global_id is 4618 2024-03-29T11:45:07.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.816+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b8003520 con 0x7f45d013f650 2024-03-29T11:45:07.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.816+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2639002821 0 0) 0x7f45b8003520 con 0x7f45d013f650 2024-03-29T11:45:07.827 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.816+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 >> v1:172.21.15.33:6790/0 conn(0x7f45d00a9d70 legacy=0x7f45d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.816+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 >> v1:172.21.15.33:6789/0 conn(0x7f45d01457f0 legacy=0x7f45d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.816+0000 7f45cd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.816+0000 7f45cd59a640 1 monclient: found mon.noname-c 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d0146340 con 0x7f45d013f650 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: _finish_auth 0 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45d4d32640 5 monclient: authenticate success, global_id 4618 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45d4d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f45c8001a10 con 0x7f45d013f650 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:07.828 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: dump: 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45d4d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c8001ca0 con 0x7f45d013f650 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 1 -- 172.21.15.33:0/1977673548 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f45c8002030 con 0x7f45d013f650 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.829 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45cd59a640 10 monclient: dump: 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45d4d32640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.817+0000 7f45d4d32640 4 set_mon_vals no callback set 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.818+0000 7f45d4d32640 10 monclient: shutdown 2024-03-29T11:45:07.830 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.818+0000 7f45d4d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.818+0000 7f45d4d32640 1 -- 172.21.15.33:0/1977673548 >> v1:172.21.15.153:6789/0 conn(0x7f45d013f650 legacy=0x7f45d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.819+0000 7f45d4d32640 1 -- 172.21.15.33:0/1977673548 shutdown_connections 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.819+0000 7f45d4d32640 1 -- 172.21.15.33:0/1977673548 >> 172.21.15.33:0/1977673548 conn(0x7f45d0066a00 msgr2=0x7f45d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.819+0000 7f45d4d32640 1 -- 172.21.15.33:0/1977673548 shutdown_connections 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.819+0000 7f45d4d32640 1 -- 172.21.15.33:0/1977673548 wait complete. 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.820+0000 7f45d4d32640 10 monclient: build_initial_monmap 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.820+0000 7f45d4d32640 10 monclient: monmap: 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:45:07.831 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:45:07.832 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:07.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.821+0000 7f45d4d32640 1 librados: starting msgr at 2024-03-29T11:45:07.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.821+0000 7f45d4d32640 1 librados: starting objecter 2024-03-29T11:45:07.832 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.821+0000 7f45d4d32640 1 Processor -- start 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.821+0000 7f45d4d32640 1 -- start start 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.821+0000 7f45d4d32640 1 librados: setting wanted keys 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.821+0000 7f45d4d32640 1 librados: calling monclient init 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.821+0000 7f45d4d32640 10 monclient: init 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 10 monclient(hunting): picked mon.noname-c con 0x7f45d00a9d70 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 10 monclient(hunting): picked mon.noname-a con 0x7f45d013f650 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 10 monclient(hunting): picked mon.noname-b con 0x7f45d01457f0 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d0070300 con 0x7f45d013f650 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d0135f00 con 0x7f45d01457f0 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00715c0 con 0x7f45d00a9d70 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:07.836 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.822+0000 7f45d4d32640 10 monclient(hunting): authenticate will time out at 1464.087304s 2024-03-29T11:45:07.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.828+0000 7f45ced9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f45d01457f0 0x7f45d01357f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:44585/0 (socket says 172.21.15.33:44585) 2024-03-29T11:45:07.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.828+0000 7f45ced9d640 1 -- 172.21.15.33:0/4079150738 learned_addr learned my addr 172.21.15.33:0/4079150738 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:07.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.828+0000 7f45beffd640 10 client.?.objecter ms_handle_connect 0x7f45d013f650 2024-03-29T11:45:07.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.828+0000 7f45beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:07.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.828+0000 7f45beffd640 10 client.?.objecter ms_handle_connect 0x7f45d01457f0 2024-03-29T11:45:07.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:07.837 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572642921 0 0) 0x7f45d0070300 con 0x7f45d013f650 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b0004eb0 con 0x7f45d013f650 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221792472 0 0) 0x7f45d0135f00 con 0x7f45d01457f0 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): my global_id is 4621 2024-03-29T11:45:07.838 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d0070300 con 0x7f45d01457f0 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985065884 0 0) 0x7f45d0070300 con 0x7f45d01457f0 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45d0135f00 con 0x7f45d01457f0 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f45c40017d0 con 0x7f45d01457f0 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): dump: 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:07.839 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:07.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.829+0000 7f45beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:07.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1263034757 0 0) 0x7f45d0135f00 con 0x7f45d01457f0 2024-03-29T11:45:07.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 >> v1:172.21.15.33:6790/0 conn(0x7f45d00a9d70 legacy=0x7f45d01de910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 >> v1:172.21.15.33:6789/0 conn(0x7f45d013f650 legacy=0x7f45d01350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:07.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:07.841 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 1 monclient: found mon.b 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00718a0 con 0x7f45d01457f0 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 10 monclient: _finish_auth 0 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 5 monclient: authenticate success, global_id 4621 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 10 monclient: _renew_subs 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d0070790 con 0x7f45d01457f0 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 10 monclient: _renew_subs 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:07.842 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d006f690 con 0x7f45d01457f0 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45d4d32640 1 librados: init done 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c4001ce0 con 0x7f45d01457f0 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.830+0000 7f45ccd99640 4 set_mon_vals no callback set 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.831+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f45c4003790 con 0x7f45d01457f0 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.831+0000 7f45beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.832+0000 7f45beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.832+0000 7f45beffd640 10 monclient: dump: 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:07.843 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.833+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f45c40038f0 con 0x7f45d01457f0 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.833+0000 7f45ce59c640 10 monclient: get_auth_request con 0x7f45b006dd50 auth_method 0 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.834+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7f45c40848c0 con 0x7f45d01457f0 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.834+0000 7f45beffd640 10 client.4621.objecter ms_dispatch 0x7f45d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.834+0000 7f45beffd640 3 client.4621.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.834+0000 7f45beffd640 3 client.4621.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:07.844 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.834+0000 7f45beffd640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-29T11:45:07.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.835+0000 7f45beffd640 10 client.4621.objecter ms_handle_connect 0x7f45b006dd50 2024-03-29T11:45:07.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.835+0000 7f45d4d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:07.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.835+0000 7f45d4d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:07.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.835+0000 7f45d4d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:07.845 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.835+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b4005080 con 0x7f45d01457f0 2024-03-29T11:45:07.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.844+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45c40501a0 con 0x7f45d01457f0 2024-03-29T11:45:07.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.844+0000 7f45beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:07.849 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:07.844+0000 7f45beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:08.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.254+0000 7f45d4d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:08.259 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.254+0000 7f45d4d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:08.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.254+0000 7f45d4d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-29T11:45:08.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.254+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45b4005330 con 0x7f45d01457f0 2024-03-29T11:45:08.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.255+0000 7f45beffd640 1 -- 172.21.15.33:0/4079150738 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7f45c40520d0 con 0x7f45d01457f0 2024-03-29T11:45:08.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.255+0000 7f45beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:08.260 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.255+0000 7f45beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:08.261 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:45:08.261 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 10 librados: watch_flush enter 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 10 librados: watch_flush exit 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 10 client.4621.objecter successfully canceled tick 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 >> v1:172.21.15.153:6816/2737028074 conn(0x7f45b006dd50 legacy=0x7f45b0070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 10 monclient: shutdown 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 >> v1:172.21.15.153:6789/0 conn(0x7f45d01457f0 legacy=0x7f45d01357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 shutdown_connections 2024-03-29T11:45:08.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 >> 172.21.15.33:0/4079150738 conn(0x7f45d0066a00 msgr2=0x7f45d00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:08.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 shutdown_connections 2024-03-29T11:45:08.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 1 -- 172.21.15.33:0/4079150738 wait complete. 2024-03-29T11:45:08.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:08.259+0000 7f45d4d32640 1 librados: shutdown 2024-03-29T11:45:08.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:15.088 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:15.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.231+0000 7f15321b5640 10 monclient: get_monmap_and_config 2024-03-29T11:45:15.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.231+0000 7f15321b5640 10 monclient: build_initial_monmap 2024-03-29T11:45:15.239 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.231+0000 7f15321b5640 10 monclient: monmap: 2024-03-29T11:45:15.239 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:15.239 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:15.239 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:45:15.233531+0000 2024-03-29T11:45:15.239 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:45:15.233531+0000 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.234+0000 7f15321b5640 1 Processor -- start 2024-03-29T11:45:15.240 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.234+0000 7f15321b5640 1 -- start start 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.234+0000 7f15321b5640 10 monclient: init 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 10 monclient(hunting): picked mon.noname-a con 0x7f152c0b4100 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 10 monclient(hunting): picked mon.noname-b con 0x7f152c13ee90 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 10 monclient(hunting): picked mon.noname-c con 0x7f152c0a9ac0 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c0af6f0 con 0x7f152c0b4100 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c0aac80 con 0x7f152c0a9ac0 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c0aaf60 con 0x7f152c13ee90 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f15321b5640 10 monclient(hunting): authenticate will time out at 1471.500405s 2024-03-29T11:45:15.241 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f152b7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f152c0b4100 0x7f152c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:53378/0 (socket says 172.21.15.33:53378) 2024-03-29T11:45:15.242 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.235+0000 7f152b7fe640 1 -- 172.21.15.33:0/282859567 learned_addr learned my addr 172.21.15.33:0/282859567 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:15.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516226267 0 0) 0x7f152c0af6f0 con 0x7f152c0b4100 2024-03-29T11:45:15.243 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1514003520 con 0x7f152c0b4100 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737591172 0 0) 0x7f152c0aac80 con 0x7f152c0a9ac0 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 10 monclient(hunting): my global_id is 4624 2024-03-29T11:45:15.244 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c0af6f0 con 0x7f152c0a9ac0 2024-03-29T11:45:15.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1397330998 0 0) 0x7f152c0af6f0 con 0x7f152c0a9ac0 2024-03-29T11:45:15.245 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.239+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 >> v1:172.21.15.33:6790/0 conn(0x7f152c13ee90 legacy=0x7f152c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:15.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 >> v1:172.21.15.33:6789/0 conn(0x7f152c0b4100 legacy=0x7f152c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:15.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:15.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:45:15.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:15.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-29T11:45:15.248 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c13e970 con 0x7f152c0a9ac0 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f15240018a0 con 0x7f152c0a9ac0 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f152a7fc640 10 monclient: dump: 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:15.249 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f15321b5640 5 monclient: authenticate success, global_id 4624 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.241+0000 7f15321b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1524001a00 con 0x7f152c0a9ac0 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f152a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f152a7fc640 1 -- 172.21.15.33:0/282859567 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1524001e50 con 0x7f152c0a9ac0 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f152a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f152a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f152a7fc640 10 monclient: dump: 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:15.250 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f15321b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.242+0000 7f15321b5640 4 set_mon_vals no callback set 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.243+0000 7f15321b5640 10 monclient: shutdown 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.243+0000 7f15321b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:15.251 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.243+0000 7f15321b5640 1 -- 172.21.15.33:0/282859567 >> v1:172.21.15.153:6789/0 conn(0x7f152c0a9ac0 legacy=0x7f152c142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:15.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.243+0000 7f15321b5640 1 -- 172.21.15.33:0/282859567 shutdown_connections 2024-03-29T11:45:15.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.243+0000 7f15321b5640 1 -- 172.21.15.33:0/282859567 >> 172.21.15.33:0/282859567 conn(0x7f152c0b35e0 msgr2=0x7f152c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:15.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.245+0000 7f15321b5640 1 -- 172.21.15.33:0/282859567 shutdown_connections 2024-03-29T11:45:15.252 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.245+0000 7f15321b5640 1 -- 172.21.15.33:0/282859567 wait complete. 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 10 monclient: build_initial_monmap 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 10 monclient: monmap: 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:45:15.253 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 1 librados: starting msgr at 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 1 librados: starting objecter 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 1 Processor -- start 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 1 -- start start 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 1 librados: setting wanted keys 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 1 librados: calling monclient init 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.246+0000 7f15321b5640 10 monclient: init 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 10 monclient(hunting): picked mon.noname-a con 0x7f152c0a9ac0 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 10 monclient(hunting): picked mon.noname-b con 0x7f152c0b4100 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:15.254 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 10 monclient(hunting): picked mon.noname-c con 0x7f152c13ee90 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:15.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c070790 con 0x7f152c0a9ac0 2024-03-29T11:45:15.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c071c20 con 0x7f152c0b4100 2024-03-29T11:45:15.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c1507a0 con 0x7f152c13ee90 2024-03-29T11:45:15.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:15.255 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.247+0000 7f15321b5640 10 monclient(hunting): authenticate will time out at 1471.512368s 2024-03-29T11:45:15.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.249+0000 7f152b7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f152c0a9ac0 0x7f152c06e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:34416/0 (socket says 172.21.15.33:34416) 2024-03-29T11:45:15.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.249+0000 7f152b7fe640 1 -- 172.21.15.33:0/3228478097 learned_addr learned my addr 172.21.15.33:0/3228478097 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:15.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.250+0000 7f1513fff640 10 client.?.objecter ms_handle_connect 0x7f152c0a9ac0 2024-03-29T11:45:15.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.250+0000 7f1513fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:15.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.251+0000 7f1513fff640 10 client.?.objecter ms_handle_connect 0x7f152c13ee90 2024-03-29T11:45:15.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.251+0000 7f1513fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:15.257 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.251+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236439316 0 0) 0x7f152c1507a0 con 0x7f152c13ee90 2024-03-29T11:45:15.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.251+0000 7f1513fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:15.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.252+0000 7f1513fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:15.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.252+0000 7f1513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:15.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.252+0000 7f1513fff640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:45:15.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.252+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1518004eb0 con 0x7f152c13ee90 2024-03-29T11:45:15.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.252+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4278090037 0 0) 0x7f1518004eb0 con 0x7f152c13ee90 2024-03-29T11:45:15.258 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.252+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f152c1507a0 con 0x7f152c13ee90 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1524001d10 con 0x7f152c13ee90 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): dump: 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.264 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746305479 0 0) 0x7f152c070790 con 0x7f152c0a9ac0 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:15.265 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1518004eb0 con 0x7f152c0a9ac0 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3719240961 0 0) 0x7f152c1507a0 con 0x7f152c13ee90 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.253+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 >> v1:172.21.15.153:6789/0 conn(0x7f152c0b4100 legacy=0x7f152c14f980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 >> v1:172.21.15.33:6789/0 conn(0x7f152c0a9ac0 legacy=0x7f152c06e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 1 monclient: found mon.c 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c071e90 con 0x7f152c13ee90 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:15.266 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f1513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 10 monclient: _renew_subs 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152c072170 con 0x7f152c13ee90 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 10 monclient: _renew_subs 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f152c06fb10 con 0x7f152c13ee90 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.255+0000 7f15321b5640 1 librados: init done 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1524003c20 con 0x7f152c13ee90 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f1513fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f1524003f30 con 0x7f152c13ee90 2024-03-29T11:45:15.267 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f1513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f1513fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f1513fff640 10 monclient: dump: 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:15.268 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f15297fa640 4 set_mon_vals no callback set 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.257+0000 7f15321b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.258+0000 7f15321b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.258+0000 7f15321b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:15.269 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.258+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c005080 con 0x7f152c13ee90 2024-03-29T11:45:15.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.259+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f1524019f00 con 0x7f152c13ee90 2024-03-29T11:45:15.271 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.260+0000 7f152b7fe640 10 monclient: get_auth_request con 0x7f151806de10 auth_method 0 2024-03-29T11:45:15.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.260+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7f1524083b80 con 0x7f152c13ee90 2024-03-29T11:45:15.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.260+0000 7f1513fff640 10 client.4613.objecter ms_dispatch 0x7f152c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:15.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.260+0000 7f1513fff640 3 client.4613.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:15.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.260+0000 7f1513fff640 3 client.4613.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:15.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.260+0000 7f1513fff640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:45:15.272 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.260+0000 7f1513fff640 10 client.4613.objecter ms_handle_connect 0x7f151806de10 2024-03-29T11:45:15.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.268+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f152404f460 con 0x7f152c13ee90 2024-03-29T11:45:15.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.268+0000 7f1513fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:15.273 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.268+0000 7f1513fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:15.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.601+0000 7f15321b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:15.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.601+0000 7f15321b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:15.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.601+0000 7f15321b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:15.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.601+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f150c005330 con 0x7f152c13ee90 2024-03-29T11:45:15.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.602+0000 7f1513fff640 1 -- 172.21.15.33:0/3228478097 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7f1524051390 con 0x7f152c13ee90 2024-03-29T11:45:15.607 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.602+0000 7f1513fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:15.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.602+0000 7f1513fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:15.608 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:45:15.608 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:15.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 10 librados: watch_flush enter 2024-03-29T11:45:15.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 10 librados: watch_flush exit 2024-03-29T11:45:15.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:45:15.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:45:15.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 >> v1:172.21.15.153:6816/2737028074 conn(0x7f151806de10 legacy=0x7f15180702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:15.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 10 monclient: shutdown 2024-03-29T11:45:15.611 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:15.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 >> v1:172.21.15.33:6790/0 conn(0x7f152c13ee90 legacy=0x7f152c150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:15.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 shutdown_connections 2024-03-29T11:45:15.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.606+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 >> 172.21.15.33:0/3228478097 conn(0x7f152c0b35e0 msgr2=0x7f152c0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:15.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.607+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 shutdown_connections 2024-03-29T11:45:15.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.607+0000 7f15321b5640 1 -- 172.21.15.33:0/3228478097 wait complete. 2024-03-29T11:45:15.612 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:15.607+0000 7f15321b5640 1 librados: shutdown 2024-03-29T11:45:15.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:22.439 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.585+0000 7fdc1b515640 10 monclient: get_monmap_and_config 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.585+0000 7fdc1b515640 10 monclient: build_initial_monmap 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.585+0000 7fdc1b515640 10 monclient: monmap: 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:45:22.586721+0000 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:45:22.586721+0000 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:45:22.591 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:22.592 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.586+0000 7fdc1b515640 1 Processor -- start 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.586+0000 7fdc1b515640 1 -- start start 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.586+0000 7fdc1b515640 10 monclient: init 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.587+0000 7fdc1b515640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.587+0000 7fdc1b515640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.587+0000 7fdc1b515640 10 monclient(hunting): picked mon.noname-c con 0x7fdc140a9ac0 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.587+0000 7fdc1b515640 10 monclient(hunting): picked mon.noname-a con 0x7fdc1413ee90 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.589+0000 7fdc1b515640 10 monclient(hunting): picked mon.noname-b con 0x7fdc140af900 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.589+0000 7fdc1b515640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc140afd10 con 0x7fdc1413ee90 2024-03-29T11:45:22.595 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.589+0000 7fdc1b515640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc140af6f0 con 0x7fdc140a9ac0 2024-03-29T11:45:22.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.589+0000 7fdc1b515640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc140aac80 con 0x7fdc140af900 2024-03-29T11:45:22.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.589+0000 7fdc1b515640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:22.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.590+0000 7fdc1b515640 10 monclient(hunting): authenticate will time out at 1478.854674s 2024-03-29T11:45:22.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.591+0000 7fdc19ab2640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fdc140af900 0x7fdc14149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:56353/0 (socket says 172.21.15.33:56353) 2024-03-29T11:45:22.596 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.591+0000 7fdc19ab2640 1 -- 172.21.15.33:0/2128463496 learned_addr learned my addr 172.21.15.33:0/2128463496 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:22.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.593+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606550658 0 0) 0x7fdc140aac80 con 0x7fdc140af900 2024-03-29T11:45:22.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.593+0000 7fdc0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:22.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.593+0000 7fdc0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:22.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.593+0000 7fdc0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:22.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.593+0000 7fdc0bfff640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:45:22.600 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.593+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc00003520 con 0x7fdc140af900 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.593+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607870265 0 0) 0x7fdc00003520 con 0x7fdc140af900 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 >> v1:172.21.15.153:6789/0 conn(0x7fdc140a9ac0 legacy=0x7fdc140adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 >> v1:172.21.15.33:6789/0 conn(0x7fdc1413ee90 legacy=0x7fdc1413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 1 monclient: found mon.noname-b 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc140aaf60 con 0x7fdc140af900 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 10 monclient: _finish_auth 0 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc0c001a10 con 0x7fdc140af900 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.594+0000 7fdc0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:22.601 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.595+0000 7fdc0bfff640 10 monclient: dump: 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.595+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc0c001ca0 con 0x7fdc140af900 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.595+0000 7fdc0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.595+0000 7fdc0bfff640 1 -- 172.21.15.33:0/2128463496 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc0c002070 con 0x7fdc140af900 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.595+0000 7fdc0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:22.602 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.595+0000 7fdc0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.595+0000 7fdc0bfff640 10 monclient: dump: 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:22.603 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:22.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.596+0000 7fdc1b515640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:45:22.605 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.596+0000 7fdc1b515640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.596+0000 7fdc1b515640 4 set_mon_vals no callback set 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.597+0000 7fdc1b515640 10 monclient: shutdown 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.597+0000 7fdc1b515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.597+0000 7fdc1b515640 1 -- 172.21.15.33:0/2128463496 >> v1:172.21.15.33:6790/0 conn(0x7fdc140af900 legacy=0x7fdc14149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.598+0000 7fdc1b515640 1 -- 172.21.15.33:0/2128463496 shutdown_connections 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.598+0000 7fdc1b515640 1 -- 172.21.15.33:0/2128463496 >> 172.21.15.33:0/2128463496 conn(0x7fdc140b35e0 msgr2=0x7fdc140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.600+0000 7fdc1b515640 1 -- 172.21.15.33:0/2128463496 shutdown_connections 2024-03-29T11:45:22.606 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.600+0000 7fdc1b515640 1 -- 172.21.15.33:0/2128463496 wait complete. 2024-03-29T11:45:22.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.601+0000 7fdc1b515640 10 monclient: build_initial_monmap 2024-03-29T11:45:22.608 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.601+0000 7fdc1b515640 10 monclient: monmap: 2024-03-29T11:45:22.608 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:22.609 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:22.609 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:45:22.609 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:45:22.609 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:22.610 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:22.610 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:22.610 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:45:22.611 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:45:22.611 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:22.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.604+0000 7fdc1b515640 1 librados: starting msgr at 2024-03-29T11:45:22.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.604+0000 7fdc1b515640 1 librados: starting objecter 2024-03-29T11:45:22.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.604+0000 7fdc1b515640 1 Processor -- start 2024-03-29T11:45:22.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.604+0000 7fdc1b515640 1 -- start start 2024-03-29T11:45:22.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.604+0000 7fdc1b515640 1 librados: setting wanted keys 2024-03-29T11:45:22.613 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.604+0000 7fdc1b515640 1 librados: calling monclient init 2024-03-29T11:45:22.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.604+0000 7fdc1b515640 10 monclient: init 2024-03-29T11:45:22.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.605+0000 7fdc1b515640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:22.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.605+0000 7fdc1b515640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:22.614 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.605+0000 7fdc1b515640 10 monclient(hunting): picked mon.noname-b con 0x7fdc140a9ac0 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:22.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.610+0000 7fdc1b515640 10 monclient(hunting): picked mon.noname-c con 0x7fdc140af900 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:22.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.610+0000 7fdc1b515640 10 monclient(hunting): picked mon.noname-a con 0x7fdc1413ee90 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:22.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.610+0000 7fdc1b515640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc14141920 con 0x7fdc1413ee90 2024-03-29T11:45:22.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.610+0000 7fdc1b515640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1413f5c0 con 0x7fdc140a9ac0 2024-03-29T11:45:22.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.610+0000 7fdc1b515640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc141422c0 con 0x7fdc140af900 2024-03-29T11:45:22.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.610+0000 7fdc1b515640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:22.615 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.610+0000 7fdc1b515640 10 monclient(hunting): authenticate will time out at 1478.874781s 2024-03-29T11:45:22.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.614+0000 7fdc18ab0640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fdc140af900 0x7fdc1413c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:56535/0 (socket says 172.21.15.33:56535) 2024-03-29T11:45:22.619 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.614+0000 7fdc18ab0640 1 -- 172.21.15.33:0/2658211898 learned_addr learned my addr 172.21.15.33:0/2658211898 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:22.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 client.?.objecter ms_handle_connect 0x7fdc140af900 2024-03-29T11:45:22.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:22.624 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 client.?.objecter ms_handle_connect 0x7fdc1413ee90 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16638843 0 0) 0x7fdc141422c0 con 0x7fdc140af900 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf8004eb0 con 0x7fdc140af900 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629085934 0 0) 0x7fdc14141920 con 0x7fdc1413ee90 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc141422c0 con 0x7fdc1413ee90 2024-03-29T11:45:22.625 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1208400106 0 0) 0x7fdbf8004eb0 con 0x7fdc140af900 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc14141920 con 0x7fdc140af900 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc10001960 con 0x7fdc140af900 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:22.626 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.615+0000 7fdc09ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 804653093 0 0) 0x7fdc14141920 con 0x7fdc140af900 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 >> v1:172.21.15.153:6789/0 conn(0x7fdc140a9ac0 legacy=0x7fdc14140240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 >> v1:172.21.15.33:6789/0 conn(0x7fdc1413ee90 legacy=0x7fdc141e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 1 monclient: found mon.c 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1413d7e0 con 0x7fdc140af900 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:45:22.627 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 10 monclient: _renew_subs 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc14140950 con 0x7fdc140af900 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc10001ac0 con 0x7fdc140af900 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fdc100038a0 con 0x7fdc140af900 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc09ffb640 10 monclient: dump: 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:22.628 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 10 monclient: _renew_subs 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1406e980 con 0x7fdc140af900 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.616+0000 7fdc1b515640 1 librados: init done 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.618+0000 7fdc0affd640 4 set_mon_vals no callback set 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.618+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fdc10003a00 con 0x7fdc140af900 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.619+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7fdc10084a40 con 0x7fdc140af900 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.619+0000 7fdc09ffb640 10 client.4619.objecter ms_dispatch 0x7fdc14000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.619+0000 7fdc09ffb640 3 client.4619.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.619+0000 7fdc09ffb640 3 client.4619.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:22.629 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.619+0000 7fdc09ffb640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T11:45:22.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.619+0000 7fdc192b1640 10 monclient: get_auth_request con 0x7fdbf806de20 auth_method 0 2024-03-29T11:45:22.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.620+0000 7fdc09ffb640 10 client.4619.objecter ms_handle_connect 0x7fdbf806de20 2024-03-29T11:45:22.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.622+0000 7fdc1b515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:22.630 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.622+0000 7fdc1b515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:22.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.622+0000 7fdc1b515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:22.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.622+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf4005080 con 0x7fdc140af900 2024-03-29T11:45:22.638 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.634+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc10050150 con 0x7fdc140af900 2024-03-29T11:45:22.639 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.634+0000 7fdc09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:22.639 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.634+0000 7fdc09ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:22.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.955+0000 7fdc1b515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:22.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.955+0000 7fdc1b515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:22.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.955+0000 7fdc1b515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:22.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.955+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbf4005330 con 0x7fdc140af900 2024-03-29T11:45:22.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.957+0000 7fdc09ffb640 1 -- 172.21.15.33:0/2658211898 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7fdc10052250 con 0x7fdc140af900 2024-03-29T11:45:22.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.958+0000 7fdc09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:22.963 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.958+0000 7fdc09ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:22.963 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:45:22.963 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:22.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.961+0000 7fdc1b515640 10 librados: watch_flush enter 2024-03-29T11:45:22.966 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.961+0000 7fdc1b515640 10 librados: watch_flush exit 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.961+0000 7fdc1b515640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.961+0000 7fdc1b515640 10 client.4619.objecter successfully canceled tick 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.962+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 >> v1:172.21.15.153:6816/2737028074 conn(0x7fdbf806de20 legacy=0x7fdbf80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.962+0000 7fdc1b515640 10 monclient: shutdown 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.962+0000 7fdc1b515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.962+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 >> v1:172.21.15.33:6790/0 conn(0x7fdc140af900 legacy=0x7fdc1413c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.962+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 shutdown_connections 2024-03-29T11:45:22.967 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.962+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 >> 172.21.15.33:0/2658211898 conn(0x7fdc140b35e0 msgr2=0x7fdc140ad1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:22.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.963+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 shutdown_connections 2024-03-29T11:45:22.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.963+0000 7fdc1b515640 1 -- 172.21.15.33:0/2658211898 wait complete. 2024-03-29T11:45:22.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:22.964+0000 7fdc1b515640 1 librados: shutdown 2024-03-29T11:45:22.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:29.797 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.946+0000 7fb637a31640 10 monclient: get_monmap_and_config 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.946+0000 7fb637a31640 10 monclient: build_initial_monmap 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.946+0000 7fb637a31640 10 monclient: monmap: 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:45:29.948287+0000 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:45:29.948287+0000 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:29.952 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:29.953 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:45:29.953 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:45:29.953 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.947+0000 7fb637a31640 1 Processor -- start 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.947+0000 7fb637a31640 1 -- start start 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.947+0000 7fb637a31640 10 monclient: init 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 10 monclient(hunting): picked mon.noname-b con 0x7fb6300b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 10 monclient(hunting): picked mon.noname-a con 0x7fb63014a700 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 10 monclient(hunting): picked mon.noname-c con 0x7fb6300ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6300aba70 con 0x7fb63014a700 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6300aecb0 con 0x7fb6300ac5a0 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6300af150 con 0x7fb6300b0770 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:29.955 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb637a31640 10 monclient(hunting): authenticate will time out at 1486.213301s 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb6357cd640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7fb6300b0770 0x7fb630149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:36762/0 (socket says 172.21.15.33:36762) 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.948+0000 7fb6357cd640 1 -- 172.21.15.33:0/681340195 learned_addr learned my addr 172.21.15.33:0/681340195 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588480180 0 0) 0x7fb6300af150 con 0x7fb6300b0770 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c003520 con 0x7fb6300b0770 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2817518917 0 0) 0x7fb61c003520 con 0x7fb6300b0770 2024-03-29T11:45:29.956 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.949+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 >> v1:172.21.15.153:6789/0 conn(0x7fb6300ac5a0 legacy=0x7fb630151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:29.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.951+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 >> v1:172.21.15.33:6789/0 conn(0x7fb63014a700 legacy=0x7fb6300abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:29.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.951+0000 7fb627fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:29.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 1 monclient: found mon.noname-b 2024-03-29T11:45:29.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:29.959 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300ac9d0 con 0x7fb6300b0770 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb637a31640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb637a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb62c001690 con 0x7fb6300b0770 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.952+0000 7fb627fff640 10 monclient: dump: 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.960 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb637a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb62c001920 con 0x7fb6300b0770 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb637a31640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb637a31640 4 set_mon_vals no callback set 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb627fff640 1 -- 172.21.15.33:0/681340195 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb62c001cf0 con 0x7fb6300b0770 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb627fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.953+0000 7fb627fff640 10 monclient: dump: 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:29.961 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.954+0000 7fb637a31640 10 monclient: shutdown 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.954+0000 7fb637a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.954+0000 7fb637a31640 1 -- 172.21.15.33:0/681340195 >> v1:172.21.15.33:6790/0 conn(0x7fb6300b0770 legacy=0x7fb630149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.954+0000 7fb637a31640 1 -- 172.21.15.33:0/681340195 shutdown_connections 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.954+0000 7fb637a31640 1 -- 172.21.15.33:0/681340195 >> 172.21.15.33:0/681340195 conn(0x7fb63006b660 msgr2=0x7fb63006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:29.962 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.954+0000 7fb637a31640 1 -- 172.21.15.33:0/681340195 shutdown_connections 2024-03-29T11:45:29.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.954+0000 7fb637a31640 1 -- 172.21.15.33:0/681340195 wait complete. 2024-03-29T11:45:29.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.955+0000 7fb637a31640 10 monclient: build_initial_monmap 2024-03-29T11:45:29.968 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.955+0000 7fb637a31640 10 monclient: monmap: 2024-03-29T11:45:29.968 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:29.968 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:29.968 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.955+0000 7fb637a31640 1 librados: starting msgr at 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.955+0000 7fb637a31640 1 librados: starting objecter 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.955+0000 7fb637a31640 1 Processor -- start 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 1 -- start start 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 1 librados: setting wanted keys 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 1 librados: calling monclient init 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient: init 2024-03-29T11:45:29.969 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient(hunting): picked mon.noname-b con 0x7fb6300ac5a0 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient(hunting): picked mon.noname-c con 0x7fb6300b0770 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient(hunting): picked mon.noname-a con 0x7fb63014a700 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6301dbbc0 con 0x7fb63014a700 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6301dc0e0 con 0x7fb6300ac5a0 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6301dc270 con 0x7fb6300b0770 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.956+0000 7fb637a31640 10 monclient(hunting): authenticate will time out at 1486.221623s 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.957+0000 7fb635fce640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fb63014a700 0x7fb6301239d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:54592/0 (socket says 172.21.15.33:54592) 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.957+0000 7fb635fce640 1 -- 172.21.15.33:0/3746355192 learned_addr learned my addr 172.21.15.33:0/3746355192 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.957+0000 7fb625ffb640 10 client.?.objecter ms_handle_connect 0x7fb63014a700 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.957+0000 7fb625ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.957+0000 7fb625ffb640 10 client.?.objecter ms_handle_connect 0x7fb6300b0770 2024-03-29T11:45:29.970 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.957+0000 7fb625ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.0 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499080689 0 0) 0x7fb6301dbbc0 con 0x7fb63014a700 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb614004eb0 con 0x7fb63014a700 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463991594 0 0) 0x7fb6301dc270 con 0x7fb6300b0770 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6301dbbc0 con 0x7fb6300b0770 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2137588478 0 0) 0x7fb6301dbbc0 con 0x7fb6300b0770 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6301dc270 con 0x7fb6300b0770 2024-03-29T11:45:29.971 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb628001ac0 con 0x7fb6300b0770 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.958+0000 7fb625ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.959+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 368895911 0 0) 0x7fb6301dc270 con 0x7fb6300b0770 2024-03-29T11:45:29.972 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.959+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 >> v1:172.21.15.153:6789/0 conn(0x7fb6300ac5a0 legacy=0x7fb6301db360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 >> v1:172.21.15.33:6789/0 conn(0x7fb63014a700 legacy=0x7fb6301239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 1 monclient: found mon.c 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630147c30 con 0x7fb6300b0770 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb625ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 10 monclient: _renew_subs 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb630147ee0 con 0x7fb6300b0770 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 10 monclient: _renew_subs 2024-03-29T11:45:29.973 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb630148330 con 0x7fb6300b0770 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.960+0000 7fb637a31640 1 librados: init done 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.962+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb628001fd0 con 0x7fb6300b0770 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.962+0000 7fb625ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.962+0000 7fb637a31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.962+0000 7fb6277fe640 4 set_mon_vals no callback set 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.962+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7fb628003990 con 0x7fb6300b0770 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.963+0000 7fb637a31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.963+0000 7fb637a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:29.974 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.963+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb630075f10 con 0x7fb6300b0770 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.963+0000 7fb625ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.963+0000 7fb625ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.963+0000 7fb625ffb640 10 monclient: dump: 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:29.975 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.963+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7fb628019960 con 0x7fb6300b0770 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.964+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7fb628083900 con 0x7fb6300b0770 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.964+0000 7fb625ffb640 10 client.4625.objecter ms_dispatch 0x7fb630000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.964+0000 7fb625ffb640 3 client.4625.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.964+0000 7fb625ffb640 3 client.4625.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.964+0000 7fb625ffb640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.964+0000 7fb6357cd640 10 monclient: get_auth_request con 0x7fb61406dd30 auth_method 0 2024-03-29T11:45:29.976 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.966+0000 7fb625ffb640 10 client.4625.objecter ms_handle_connect 0x7fb61406dd30 2024-03-29T11:45:29.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.971+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb62804f1e0 con 0x7fb6300b0770 2024-03-29T11:45:29.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.971+0000 7fb625ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:29.977 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:29.971+0000 7fb625ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:30.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.301+0000 7fb637a31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:30.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.301+0000 7fb637a31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:30.307 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.301+0000 7fb637a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:30.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.301+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6301dbd50 con 0x7fb6300b0770 2024-03-29T11:45:30.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.302+0000 7fb625ffb640 1 -- 172.21.15.33:0/3746355192 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7fb628051110 con 0x7fb6300b0770 2024-03-29T11:45:30.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.302+0000 7fb625ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:30.308 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.302+0000 7fb625ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:30.308 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:45:30.308 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:30.310 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.305+0000 7fb637a31640 10 librados: watch_flush enter 2024-03-29T11:45:30.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.308+0000 7fb637a31640 10 librados: watch_flush exit 2024-03-29T11:45:30.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.308+0000 7fb637a31640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T11:45:30.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.308+0000 7fb637a31640 10 client.4625.objecter successfully canceled tick 2024-03-29T11:45:30.314 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.308+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 >> v1:172.21.15.153:6816/2737028074 conn(0x7fb61406dd30 legacy=0x7fb6140701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.308+0000 7fb637a31640 10 monclient: shutdown 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.308+0000 7fb637a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.308+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 >> v1:172.21.15.33:6790/0 conn(0x7fb6300b0770 legacy=0x7fb630147520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.309+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 shutdown_connections 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.309+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 >> 172.21.15.33:0/3746355192 conn(0x7fb63006b660 msgr2=0x7fb6300abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.309+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 shutdown_connections 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.309+0000 7fb637a31640 1 -- 172.21.15.33:0/3746355192 wait complete. 2024-03-29T11:45:30.315 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:30.309+0000 7fb637a31640 1 librados: shutdown 2024-03-29T11:45:30.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:37.141 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:37.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.271+0000 7f586727b640 10 monclient: get_monmap_and_config 2024-03-29T11:45:37.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.271+0000 7f586727b640 10 monclient: build_initial_monmap 2024-03-29T11:45:37.276 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.271+0000 7f586727b640 10 monclient: monmap: 2024-03-29T11:45:37.276 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:45:37.272025+0000 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:45:37.272025+0000 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-b 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:37.277 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.272+0000 7f586727b640 1 Processor -- start 2024-03-29T11:45:37.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.272+0000 7f586727b640 1 -- start start 2024-03-29T11:45:37.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.272+0000 7f586727b640 10 monclient: init 2024-03-29T11:45:37.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:37.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:37.278 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5865017640 1 --1- >> v1:172.21.15.33:6790/0 conn(0x7f58600a9750 0x7f58601436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6790/0 says I am v1:172.21.15.33:38384/0 (socket says 172.21.15.33:38384) 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 10 monclient(hunting): picked mon.noname-b con 0x7f58600a9750 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5865017640 1 -- 172.21.15.33:0/1924994189 learned_addr learned my addr 172.21.15.33:0/1924994189 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 10 monclient(hunting): picked mon.noname-c con 0x7f5860149ac0 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 10 monclient(hunting): picked mon.noname-a con 0x7f586014fe50 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600aa4a0 con 0x7f586014fe50 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586014a680 con 0x7f5860149ac0 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600af290 con 0x7f58600a9750 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f586727b640 10 monclient(hunting): authenticate will time out at 1493.537377s 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740257551 0 0) 0x7f58600af290 con 0x7f58600a9750 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5853fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5853fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5853fff640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:45:37.279 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.273+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f584c003520 con 0x7f58600a9750 2024-03-29T11:45:37.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.274+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805444687 0 0) 0x7f584c003520 con 0x7f58600a9750 2024-03-29T11:45:37.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.274+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 >> v1:172.21.15.153:6789/0 conn(0x7f5860149ac0 legacy=0x7f5860149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:37.280 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.274+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 >> v1:172.21.15.33:6789/0 conn(0x7f586014fe50 legacy=0x7f5860152240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:37.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:37.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 1 monclient: found mon.noname-b 2024-03-29T11:45:37.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:37.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.33:6790/0 2024-03-29T11:45:37.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600afb40 con 0x7f58600a9750 2024-03-29T11:45:37.281 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f585c001690 con 0x7f58600a9750 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f586727b640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: dump: 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.282 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f586727b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 <== mon.2 v1:172.21.15.33:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f585c001860 con 0x7f58600a9750 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 1 -- 172.21.15.33:0/1924994189 <== mon.2 v1:172.21.15.33:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f585c001cb0 con 0x7f58600a9750 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f5853fff640 10 monclient: dump: 2024-03-29T11:45:37.283 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f586727b640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:37.284 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.277+0000 7f586727b640 4 set_mon_vals no callback set 2024-03-29T11:45:37.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.278+0000 7f586727b640 10 monclient: shutdown 2024-03-29T11:45:37.289 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.278+0000 7f586727b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.278+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 >> v1:172.21.15.33:6790/0 conn(0x7f58600a9750 legacy=0x7f58601436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.278+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 shutdown_connections 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.278+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 >> 172.21.15.33:0/1924994189 conn(0x7f58600b35e0 msgr2=0x7f58600b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.278+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 shutdown_connections 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.278+0000 7f586727b640 1 -- 172.21.15.33:0/1924994189 wait complete. 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 10 monclient: build_initial_monmap 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 10 monclient: monmap: 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:epoch 0 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:last_changed 0.000000 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:created 0.000000 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:37.290 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.noname-a 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.noname-c 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 1 librados: starting msgr at 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 1 librados: starting objecter 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 1 Processor -- start 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 1 -- start start 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 1 librados: setting wanted keys 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 1 librados: calling monclient init 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.279+0000 7f586727b640 10 monclient: init 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 10 monclient(hunting): picked mon.noname-b con 0x7f58600a9750 addr v1:172.21.15.153:6789/0 2024-03-29T11:45:37.291 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 10 monclient(hunting): picked mon.noname-a con 0x7f5860149ac0 addr v1:172.21.15.33:6789/0 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 10 monclient(hunting): picked mon.noname-c con 0x7f586014fe50 addr v1:172.21.15.33:6790/0 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58601df6f0 con 0x7f5860149ac0 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5860145bf0 con 0x7f58600a9750 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 1 -- --> v1:172.21.15.33:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5860145d80 con 0x7f586014fe50 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f586727b640 10 monclient(hunting): authenticate will time out at 1493.544297s 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f5865017640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f58600a9750 0x7f58601dee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.33:43094/0 (socket says 172.21.15.33:43094) 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.280+0000 7f5865017640 1 -- 172.21.15.33:0/3541775689 learned_addr learned my addr 172.21.15.33:0/3541775689 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 client.?.objecter ms_handle_connect 0x7f586014fe50 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 client.?.objecter ms_handle_connect 0x7f58600a9750 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480583712 0 0) 0x7f5860145d80 con 0x7f586014fe50 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:37.292 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.33:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5844004eb0 con 0x7f586014fe50 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229677630 0 0) 0x7f5860145bf0 con 0x7f58600a9750 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): my global_id is 4630 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5860145d80 con 0x7f58600a9750 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2639717063 0 0) 0x7f5844004eb0 con 0x7f586014fe50 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.33:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5860145bf0 con 0x7f586014fe50 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5858001950 con 0x7f586014fe50 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:37.293 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2228842247 0 0) 0x7f5860145d80 con 0x7f58600a9750 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5844004eb0 con 0x7f58600a9750 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f585c001b70 con 0x7f58600a9750 2024-03-29T11:45:37.294 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.281+0000 7f5851ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.282+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1067671506 0 0) 0x7f5860145bf0 con 0x7f586014fe50 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.282+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 >> v1:172.21.15.153:6789/0 conn(0x7f58600a9750 legacy=0x7f58601dee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:37.295 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.282+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 >> v1:172.21.15.33:6789/0 conn(0x7f5860149ac0 legacy=0x7f586013d3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 1 monclient: found mon.c 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.33:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586013db00 con 0x7f586014fe50 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f586727b640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5858001ea0 con 0x7f586014fe50 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f586727b640 10 monclient: _renew_subs 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f586727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.33:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586013dd50 con 0x7f586014fe50 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:37.296 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 361960754 0 0) 0x7f5858003db0 con 0x7f586014fe50 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f5851ffb640 10 monclient: dump: 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:epoch 1 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:fsid 77cf7d38-28bf-40b4-b03e-b6d79f38b4e6 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:last_changed 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:created 2024-03-29T11:34:00.510811+0000 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:election_strategy: 1 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:0: v1:172.21.15.33:6789/0 mon.a 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:2: v1:172.21.15.33:6790/0 mon.c 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr: 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.283+0000 7f58537fe640 4 set_mon_vals no callback set 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.284+0000 7f586727b640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:37.297 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.284+0000 7f586727b640 10 monclient: _renew_subs 2024-03-29T11:45:37.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.284+0000 7f586727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:37.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.284+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.33:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58600ac5d0 con 0x7f586014fe50 2024-03-29T11:45:37.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.284+0000 7f586727b640 1 librados: init done 2024-03-29T11:45:37.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.285+0000 7f586727b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:37.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.285+0000 7f586727b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:37.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.285+0000 7f586727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:37.298 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.285+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5860075f10 con 0x7f586014fe50 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.286+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2734193672 0 0) 0x7f5858019d80 con 0x7f586014fe50 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.287+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442995205 0 0) 0x7f5858083c00 con 0x7f586014fe50 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.287+0000 7f5851ffb640 10 client.4631.objecter ms_dispatch 0x7f5860000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.287+0000 7f5851ffb640 3 client.4631.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.287+0000 7f5851ffb640 3 client.4631.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.287+0000 7f5865017640 10 monclient: get_auth_request con 0x7f584406ddf0 auth_method 0 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.287+0000 7f5851ffb640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:45:37.299 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.288+0000 7f5851ffb640 10 client.4631.objecter ms_handle_connect 0x7f584406ddf0 2024-03-29T11:45:37.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.294+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f585804f540 con 0x7f586014fe50 2024-03-29T11:45:37.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.294+0000 7f5851ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:37.300 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.294+0000 7f5851ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:37.622 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.617+0000 7f586727b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:37.622 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.617+0000 7f586727b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:37.622 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.617+0000 7f586727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.33:6790/0 2024-03-29T11:45:37.622 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.618+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 --> v1:172.21.15.33:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f586014a9f0 con 0x7f586014fe50 2024-03-29T11:45:37.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.624+0000 7f5851ffb640 1 -- 172.21.15.33:0/3541775689 <== mon.2 v1:172.21.15.33:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4135134085) 0x7f58580832b0 con 0x7f586014fe50 2024-03-29T11:45:37.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.624+0000 7f5851ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:37.628 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.624+0000 7f5851ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:37.628 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-29T11:45:37.628 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":11,"fsid":"77cf7d38-28bf-40b4-b03e-b6d79f38b4e6","created":"2024-03-29T11:34:38.649483+0000","modified":"2024-03-29T11:44:43.575642+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:43.575642+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dcbfff5-b024-40da-ae53-b95e8ef368d6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8966becd-6685-4b87-90fa-332cb668e814","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b574eda-e056-425d-9498-efa05d611207","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d95fd6fd-9fcb-419f-9953-c0d0fa896631","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"355b699b-9a4c-4f8d-a4d0-41e3ad19559a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"057e8b01-3785-482b-8261-8b53a0386027","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"17b518fe-d9df-4355-93b9-bc2f826b30cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b7280579-d574-43a9-a294-0b40c1b3b700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:37.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 10 librados: watch_flush enter 2024-03-29T11:45:37.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 10 librados: watch_flush exit 2024-03-29T11:45:37.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:45:37.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:45:37.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 >> v1:172.21.15.153:6816/2737028074 conn(0x7f584406ddf0 legacy=0x7f58440702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:37.631 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 10 monclient: shutdown 2024-03-29T11:45:37.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:37.632 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.627+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 >> v1:172.21.15.33:6790/0 conn(0x7f586014fe50 legacy=0x7f58601db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:37.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.628+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 shutdown_connections 2024-03-29T11:45:37.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.628+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 >> 172.21.15.33:0/3541775689 conn(0x7f58600b35e0 msgr2=0x7f58600b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:37.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.628+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 shutdown_connections 2024-03-29T11:45:37.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.628+0000 7f586727b640 1 -- 172.21.15.33:0/3541775689 wait complete. 2024-03-29T11:45:37.634 INFO:teuthology.misc.health.smithi033.stderr:2024-03-29T11:45:37.628+0000 7f586727b640 1 librados: shutdown 2024-03-29T11:45:37.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:37.649 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:45:37.651 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:45:37.651 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:45:37.651 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:37.679 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:45:37.679 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:45:37.679 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:37.758 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:45:37.759 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:45:37.759 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:37.828 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:45:37.828 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:45:37.828 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:37.904 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:45:37.904 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:45:37.904 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:37.955 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:45:37.956 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:45:37.956 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:37.988 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:45:37.988 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:45:37.988 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:38.056 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:45:38.057 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:45:38.057 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:38.123 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:45:38.123 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:45:38.123 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:45:38.123 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:38.147 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:45:38.147 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:45:38.147 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:38.189 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:45:38.189 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:45:38.189 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:45:38.189 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:38.201 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:45:38.201 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:45:38.201 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:38.252 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:45:38.252 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:45:38.252 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:38.263 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:45:38.263 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:45:38.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:45:38.323 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:45:38.323 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:45:40.796 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:45:40.796 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:45:40.869 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:45:40.869 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:45:40.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:45:40.926 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:45:40.975 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:45:40.975 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:45:42.434 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:45:42.434 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:45:42.506 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:45:42.506 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:45:42.570 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:45:42.571 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:45:42.629 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:45:42.631 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/data/mon.a.tgz 2024-03-29T11:45:42.631 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-29T11:45:42.650 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.F2bOJD7kTB 2024-03-29T11:45:42.650 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.F2bOJD7kTB 2024-03-29T11:45:42.773 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.F2bOJD7kTB 2024-03-29T11:45:42.861 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.F2bOJD7kTB is 211KB 2024-03-29T11:45:42.957 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.F2bOJD7kTB 2024-03-29T11:45:42.972 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/data/mon.c.tgz 2024-03-29T11:45:42.972 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-29T11:45:43.026 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.Qd2pgstKqF 2024-03-29T11:45:43.026 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Qd2pgstKqF 2024-03-29T11:45:43.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.Qd2pgstKqF 2024-03-29T11:45:43.212 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.Qd2pgstKqF is 211KB 2024-03-29T11:45:43.307 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.Qd2pgstKqF 2024-03-29T11:45:43.321 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/data/mon.b.tgz 2024-03-29T11:45:43.321 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-29T11:45:43.337 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.NY2WjUhhzi 2024-03-29T11:45:43.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NY2WjUhhzi 2024-03-29T11:45:43.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.NY2WjUhhzi 2024-03-29T11:45:43.535 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.NY2WjUhhzi is 211KB 2024-03-29T11:45:43.630 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.NY2WjUhhzi 2024-03-29T11:45:43.644 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:45:43.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:45:43.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:45:43.727 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:45:43.766 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:45:43.766 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:45:43.768 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/remote/smithi033/crash 2024-03-29T11:45:43.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:45:43.796 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/remote/smithi153/crash 2024-03-29T11:45:43.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:45:43.832 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:45:43.832 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:45:43.840 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:45:43.910 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-03-29T11:45:43.910 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:45:43.910 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40508.log 2024-03-29T11:45:43.910 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39592.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38028.log 2024-03-29T11:45:43.910 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40508.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38772.log 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40508.log.gz 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38264.log 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38186.log 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38028.log.gz 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40235.log 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38772.log.gz 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38186.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38264.log.gz 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41288.log 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40235.log: gzip -5 --verbose -- 86.7% /var/log/ceph/ceph-client.admin.41171.log -- replaced with /var/log/ceph/ceph-client.admin.38186.log.gz 2024-03-29T11:45:43.911 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40235.log.gz 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38147.log 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41288.log.gz 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39845.log 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38577.log 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38147.log.gz 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38577.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39845.log.gz 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-03-29T11:45:43.912 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39762.log 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38577.log.gz 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38067.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40664.log 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38889.log 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 85.7% 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40664.log.gz 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38889.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37989.log 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38889.log.gz 2024-03-29T11:45:43.913 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37989.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38967.log 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37989.log.gz 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39279.log 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38967.log.gz 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38381.log 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-03-29T11:45:43.914 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39279.log.gz 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38655.log 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39709.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38381.log.gz 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38655.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40898.log 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38655.log.gz 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39045.log 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40898.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2024-03-29T11:45:43.915 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40196.log 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40586.log 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40898.log.gz 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40781.log 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39045.log.gz 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40586.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40196.log.gz 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39123.log 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40430.log 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40586.log.gz 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39123.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40781.log.gz 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40469.log 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39123.log.gz 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40001.log 2024-03-29T11:45:43.916 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40430.log: /var/log/ceph/ceph-client.admin.40469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41015.log 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40430.log.gz 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40469.log.gz 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39884.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40001.log.gz 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38342.log 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41093.log 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41015.log.gz 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-03-29T11:45:43.917 INFO:teuthology.orchestra.run.smithi033.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38342.log.gz 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41093.log.gz 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38616.log 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38811.log 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39962.log 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38811.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38616.log.gz 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38225.log 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38811.log.gz 2024-03-29T11:45:43.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.3.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39962.log.gz 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38303.log 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38499.log 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38225.log.gz 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38303.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37950.log 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38499.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38303.log.gz 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38850.log 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38499.log.gz 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39801.log 2024-03-29T11:45:43.919 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.37950.log.gz 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38850.log: gzip -5 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39801.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38850.log.gz 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38460.log 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34559.log 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39084.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39801.log.gz 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34268.log 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38460.log.gz 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 2024-03-29T11:45:43.920 INFO:teuthology.orchestra.run.smithi033.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34559.log.gz 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39435.log 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38538.log 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40742.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.34268.log.gz 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39631.log 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38538.log: gzip 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39435.log.gz -5 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41327.log 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39631.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38538.log.gz 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39006.log 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39631.log.gz 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28461.log 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41327.log.gz 2024-03-29T11:45:43.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37872.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.tmp-client.admin.28461.log: gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28461.log.gz 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37911.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39006.log.gz 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.37911.log: gzip 86.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.41210.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37872.log.gz 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.37911.log.gz 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.41210.log: -5 --verbose 87.2% -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41054.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose/var/log/ceph/ceph-osd.0.log: -- /var/log/ceph/ceph-client.admin.40274.log 2024-03-29T11:45:43.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38694.log 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40274.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41054.log.gz 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41132.log 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40274.log.gz 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5/var/log/ceph/ceph-client.admin.38694.log: --verbose -- /var/log/ceph/ceph-client.admin.34662.log 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41132.log: gzip -5 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38694.log.gz 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41132.log.gzgzip 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38733.log 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34662.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34662.log.gz 2024-03-29T11:45:43.923 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40859.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38733.log.gz 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log/var/log/ceph/ceph-client.admin.39162.log: 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39240.log 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-29T11:45:43.924 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39240.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gz 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40391.log 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40391.log: 93.8%gzip -- replaced with /var/log/ceph/ceph-mgr.y.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.39396.log 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: 86.1%/var/log/ceph/ceph-client.admin.39201.log: -- replaced with /var/log/ceph/ceph-client.admin.40391.log.gz 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40313.log 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gzgzip 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38420.log 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39396.log: /var/log/ceph/ceph-client.admin.40313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.38420.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40313.log.gz 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39396.log.gz 2024-03-29T11:45:43.925 INFO:teuthology.orchestra.run.smithi033.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38420.log.gz 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40703.log 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40352.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40703.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38106.log 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40703.log.gz 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.38106.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38106.log.gz 2024-03-29T11:45:43.926 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40547.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2024-03-29T11:45:43.931 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:45:43.931 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34575.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34695.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34575.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34695.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34575.log.gz 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34495.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34695.log.gz 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34415.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34495.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34415.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34495.log.gz 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34455.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34415.log.gz 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:45:43.932 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34535.log 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34455.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34615.log 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8%/var/log/ceph/ceph-client.admin.34535.log: -- replaced with /var/log/ceph/ceph-client.admin.34455.log.gz 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34615.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34535.log.gz 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34615.log.gz 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.34655.log: /var/log/ceph/ceph-mon.b.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 2024-03-29T11:45:43.933 INFO:teuthology.orchestra.run.smithi153.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:45:43.955 INFO:teuthology.orchestra.run.smithi153.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:45:55.304 INFO:teuthology.orchestra.run.smithi153.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:45:55.426 INFO:teuthology.orchestra.run.smithi153.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:45:59.092 INFO:teuthology.orchestra.run.smithi153.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:45:59.393 INFO:teuthology.orchestra.run.smithi153.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:46:01.925 INFO:teuthology.orchestra.run.smithi033.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:46:01.950 INFO:teuthology.orchestra.run.smithi033.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:46:01.983 INFO:teuthology.orchestra.run.smithi033.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:46:02.117 INFO:teuthology.orchestra.run.smithi033.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:46:30.993 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:46:32.712 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:46:32.714 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-29T11:46:32.714 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m48.862s 2024-03-29T11:46:32.714 INFO:teuthology.orchestra.run.smithi033.stderr:user 2m42.659s 2024-03-29T11:46:32.714 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m5.471s 2024-03-29T11:46:52.062 INFO:teuthology.orchestra.run.smithi153.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:46:52.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-29T11:46:52.064 INFO:teuthology.orchestra.run.smithi153.stderr:real 1m8.172s 2024-03-29T11:46:52.064 INFO:teuthology.orchestra.run.smithi153.stderr:user 1m57.204s 2024-03-29T11:46:52.064 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m4.333s 2024-03-29T11:46:52.065 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:46:52.065 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/remote/smithi033/log 2024-03-29T11:46:52.066 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:15.932 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/remote/smithi153/log 2024-03-29T11:47:15.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:36.639 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:36.843 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=785dced338fd473d822eee08c6ae8b3a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:36.847 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:47:36.855 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:47:36.865 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:47:36.865 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:36.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:36.961 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:47:36.970 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:47:36.970 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:36.973 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:36.987 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:36.990 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:36.992 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:36.992 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-29T11:47:36.993 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +394us[ +394us] +/- 72ms 2024-03-29T11:47:36.993 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -138us[ -149us] +/- 68ms 2024-03-29T11:47:36.993 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 -243us[ -254us] +/- 67ms 2024-03-29T11:47:36.993 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:36.994 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:36.995 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-29T11:47:36.995 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +401us[ +401us] +/- 72ms 2024-03-29T11:47:36.995 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -138us[ -134us] +/- 68ms 2024-03-29T11:47:36.995 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -248us[ -248us] +/- 67ms 2024-03-29T11:47:36.995 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:36.996 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:47:37.005 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:47:37.006 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:47:37.016 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:37.104 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:37.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:37.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:37.238 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:47:37.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:47:37.255 INFO:teuthology.task.internal:Duration was 1283.434447 seconds 2024-03-29T11:47:37.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:47:37.264 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:47:37.264 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:37.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:37.309 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:37.324 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:37.560 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:47:37.560 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-03-29T11:47:37.561 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:37.612 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-29T11:47:37.612 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:37.637 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:47:37.638 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:37.655 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:37.707 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:47:37.708 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:37.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:37.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:47:37.860 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:47:37.860 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:37.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:37.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:47:37.928 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:37.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:37.953 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-03-29T11:47:37.981 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-29T11:47:38.010 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:38.037 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:38.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:47:38.089 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:47:38.095 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/remote/smithi033 2024-03-29T11:47:38.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:38.165 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711/remote/smithi153 2024-03-29T11:47:38.165 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:38.207 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:47:38.207 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:38.211 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:38.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:47:38.289 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:47:38.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:47:38.315 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:47:38.316 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:38.318 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:38.330 INFO:teuthology.orchestra.run.smithi033.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:38.333 INFO:teuthology.orchestra.run.smithi153.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:38.334 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:47:38.373 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:47:38.461 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:47:38.493 DEBUG:teuthology.nuke:shortname: smithi033 2024-03-29T11:47:38.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:38.532 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-29T11:47:38.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:38.548 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.360207', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLfB6oBFzuCD9lJafl5tdfIzT0/P1jWSao7uthDO+9x9yuQW/zyI15ZyOrti/J982iJuh3X09GkUwuH85bqbgg='} 2024-03-29T11:47:38.574 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:37.359037', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUFjPwIOVgPGNqn7AKOiSDnNg4zyeWjB81qwLpb6ObHbse3OpFmOyuwE9sNtnpAqwNhxHi5RyHAklknpOIBC7s='} 2024-03-29T11:47:38.620 INFO:teuthology.orchestra.console.smithi033:Power off 2024-03-29T11:47:38.620 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:38.653 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-29T11:47:38.654 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:38.667 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:38.667 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:38.686 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:38.686 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:38.725 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-29T11:47:38.739 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-29T11:47:42.727 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:42.741 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:42.839 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-29T11:47:42.852 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-29T11:47:46.842 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:46.853 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:46.953 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2024-03-29T11:47:46.965 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-29T11:47:47.055 INFO:teuthology.orchestra.console.smithi033:Power off completed 2024-03-29T11:47:47.066 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-29T11:47:47.171 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1283.4344465732574 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=785dced338fd473d822eee08c6ae8b3a status: fail success: false 2024-03-29T11:47:47.171 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:47:47.249 INFO:teuthology.run:FAIL